Okay, now in this tutorial, we will cover how to profile your NeuroMechFly simulation,
namely how to understand which part of the simulation is taking a lot of time.
We'll start by profiling your simulation on the CPU.
If you have cloned the FlyGym repository, you'll find in the scripts folder
replay_behaviour_cpu[.py] and replay_behaviour_gpu[.py].
Essentially, these are the same behavior replay task
from the previous tutorials, but now let's say integrated as a single script
that you can simply run.
You can run the simulations outside of a Jupyter notebook and save
the output data to a certain folder.
Here you will see basically the same joint-angle visualization observation data over time,
pose over time, and the replay video in this folder.
And if you want to profile the performance of it without any external tool,
you can already use MuJoCo's internal profiling tool
by adding this flag to the end of the script.
Very nicely, MoJoCo will you that...
for example, you're spending
this many milliseconds in total or this many microseconds per step
computing the positions of different vertices on your meshes.
And this amount of time is spent on collision detection,
so on and so forth.
Now, if we do use an external tool,
namely py-spy,
a profiler for Python program, you can get this kind of result.
So now you'll find this new file, speedscope.json, which contains more detailed information on
not just MuJoCo,
but everything that has happened during this simulation. So I'm actually running this on a remote
workstation. So let me start by copying it to my local computer.
And the nice thing about it is that you can go to this website called speedscope.app,
which helps visualize the data that has been recorded. So
simply by dragging it into this browser, you can see something like this. I think I've
sent over the course of two or maybe three seconds,
you can zoom into a small maybe 0.1 second each window and see what's taking
the simulation time.
So for example, here you'll see that in a set_actuat[or]_inputs, it took
two milliseconds, followed by step. But within step, you have the different steps, solving for
kinematics, etc. etc., constraints...
And then you have this render_as_needed function.
Typically, probably it doesn't render anything, or maybe in this case it did.
But in most cases when you call render as needed, depending on whether that frame
should actually be rendered, it probably doesn't do much.
Similarly, when you call get_joint_angles, you can see how much time it's taking and
this kind of information.
The thing that is really useful is that, let's say,
if in my simulation I'm also doing something else,
for example, let's say my simulation within the loop.
Let's say for some reason I decided to create a thousand Python lists within each iteration.
And we run this again.
That's not efficient enough.
Maybe we create
a thousand lists of a thousand elements just to emulate something that might be not so
efficient in our code.
OK, that's how much we don't have time for that.
And once again, we can put this in this visualizer.
And now you can see that
you have these steps, but also after steps, there's a bunch of weird stuff
that's probably our list creation and copying things that's taking a lot of time.
If you go to this left heavy part, it's going to show you
exactly which part of it. Instead of sorting it sequentially over
time, it's going to tell you this much time.
Five seconds is spent doing this thing.
Of course, here it's not very informative because it's just memory addresses for the lines of code.
But if we, let's say, define a...
inefficient_stuff
It's going to tell you that out of your seven, not even seven seconds simulation,
you waste all your time on this.
In summary, the MuJoCo native timing is very good at telling you which part of the
physics simulation is taking a lot of time.
MuJoCo is super optimized.
It's probably not doing something ridiculous like the thing that we are doing.
The profiling tool with py-spy tells you what kind of operation
from your own code outside of MuJoCo costs.
It's taking a lot of time,
so it's more useful for spotting inefficiencies from your own code.
We can do something similar with the GPU.
Very similar script. Okay,
you don't have to do profiling.
You can just run this quickly to generate, to replicate the tutorials, the earlier tutorials.
But let's say you do want to profile something, you can run a script like this.
Maybe while it runs, I can explain that to view the GPU profiling output, we actually
need to use this NVIDIA Nsight Systems.
And because it is a GUI, I'm actually just using this remote desktop to view it.
Okay, so the result has been saved to flygym_outputs/gpu_profile[.nsys_rep].
You can drag and drop this file into the Nsight viewer and see what's happening.
So probably during those eight seconds, it's doing some setup steps
and those 10 seconds, it's actually running the simulation.
And you can see things like, okay, now there's a graph launch that's taking some time.
And then there's more presumably computation
where you have different lower level GPU operations.
Now, of course, it's a little bit harder to interpret GPU computation, especially if we're
doing a graph capture.
But at least you can see that there's a lot of time spent
launching the CUDA graph.
And then there's some time spent copying things probably related to copying the target joint positions.
And I mean, you can dig deeper,
but this is the kind of things that you can get
out of this kind of profiling.
Okay, that's all.
I hope this has been helpful.
If you have any questions, feel free to leave an issue under the issues page or
post a thread on the discussion board.
Thank you.