Welcome to the third NeuroMechFly video tutorial. In this tutorial, we'll cover how to compose
a simulation with a fly and maybe a fly in a certain environment using the
Python interface for NeuroMechFly.
Specifically, we're going to follow the code tutorial notebooks available on the website here.
Again, the current version of FlyGym is 2.1.0. Future versions might deviate from this video
tutorial, so please always follow the tutorials on the webpage instead of following what I'm doing
verbatim in this video.
We have already covered four different ways to install FlyGym in the previous tutorial. Since we
don't have a lot of computation going on in just composing the model, and we don't
really care about performance, I'm just going to use Google Colab.
I have already run the cell that installs FlyGym.
You may need to start by connecting to an instance and clicking this button,
and it might take some time.
So, composing models and scenes. Basically, models in MuJoCo, the underlying physics simulation for FlyGym, has
a format called MJCF, MuJoCo Modeling Format, that defines what's being simulated.
Maybe we should clarify that the word model is a little bit ambiguous. We kind of
use it interchangeably with simulation,
but in the context of MuJoCo, MuJoCo model refers to literally the
set of things that have been simulated and their physical properties like surface friction.
It does not contain dynamical information like forces and positions that you would read out when
you actually run the simulation.
The composition workflow of FlyGym is a high level wrapper around MJCF.
And it is exposed through FlyGym.compose. In this case, we start by initializing a fly.
So we create an empty NeuroMechFly model, very straightforward. You can give it a name
by default as NeuroMechFly.
And now we have an empty fly model with nothing attached to it. So next we
want to add some joints to it. And to add the joints, we need to have
two pieces of information.
First, we need to know what joints there are. This involves both knowing which anatomical joints
there are. For example, is there a joint between those two leg segments? Is there a
joint between the tibia and tarsus of a leg,
but also joint degrees of freedom? So for example, is there is the joint between tibia
and tarsus a ball joint where you can rotate in all three about all three axes?
We know this is not true. In fact, it's a one degree of freedom joint where
it acts more like a hinge instead of a ball joint.
So this kind of information needs to be defined. Another thing that needs to be defined
is the convention on the ordering of different rotational axes.
If you want to go deeper into this mathematically, the reason why this is important is
that 3D rotations, as you can see on this Wikipedia page, 3D rotations do not commute.
So the ordering of them is actually important.
Additionally, once we have decided which joints we want to add, we also want to add
joint articulation, meaning that we find a subset of joints that we want to actively control
as in actively apply forces to instead of using only their passive properties like stiffness.
We also want to define a neutral pose for the fly. So in order to do
all of this, we want to implement a couple of things from flygym.anatomy.
We have a lot of presets. So for most simulations, you can simply say I want
the axis order to be roll first, pitch next, and yaw last. We want the joints
to be the set of all biological joints. Biological meaning that, for example, if we look
at the tibia-tarsus joint, it is physically possible to have three rotational degrees of freedom,
whereas biologically, the biological
reality is that there's only one. From there, we can set up a skeleton
of the fly. So we want to use the preset of joints, all biological joints, and
this particular axis order.
Next, we can set up a neutral pose. We already have some presets. But you can
also supply a dictionary of different joint angles.
Once we have the skeleton and the neutral pose set up, we can basically add them
to the fly. I'll skip the cell block for now because it's not so crucial.
To add the joints, we use the fly.add_joint method with a specification of the skeleton
as well as the neutral pose.
The cell block basically goes back to my point that 3D rotations do not commute.
The ordering is important.
So if we want to get the same, if we want to
express the same neutral pose in roll yaw pitch versus pitch
roll yaw ordering, we actually get different numbers.
Now we have created an empty fly and added joints to it. The next step
is to add actuation to a subset of the joints.
Once again, under FlyGym Anatomy,
we have provided a bunch of presets of actuated degrees of freedom,
namely all, which means all articulated joints,
in this case all biological joints, will be actuated. Another option is legs only,
so I'll activate, not activate, but actuate active actuators to all joints that are on the legs.
And there's also another option called legs active only.
So the distinction between legs only and legs active only
is that there are further joints on the legs that are not actively actuated, biologically speaking.
So if we look at the tarsus joints, so tarsus is this last segment, you'll see
that there are joints between different tarsal segments, but the biological reality is that it's soft.
It has several segments, but every segment is so small that they are just quite compliant.
So one thing that we can do if we had selected legs only is that these
joints would be actuated as well, but a better way to do it is to use
legs active only, which would give you only actuators at the other joints, but not here.
It's important to note that maybe the biological reality is not so black and white, because
in fact there's a tendon, the long tendon, that goes from the tip of the tarsus
all the way up.
So in a way there is active actuation, but the actuation of this joint and these
joints, it's just a little bit complicated.
Anyway, in this case we choose legs active only.
The actuated degrees of freedom preset is a filter.
It depends on what joints exist in the first place when we added the joints.
So we actually have to call this get_actuated_dofs[_from_preset] method
on the skeleton object.
So if we run it, we say, okay, previously we have added all biological joints to
the skeleton and among these joints we want to actuate all joints that are on the
legs that are supposed to be active.
And it's going to tell us that we have 42 actuated degrees of freedom, which is
correct because the fly has seven of them per leg and six legs.
We can even run this to see a full list of them.
If we just look at the left front legs, those are the seven actuated joints or
degrees of freedom.
Body segments and joints and everything are semantically named.
So the joints, for example, have their names have two parts separated by a hyphen.
The first part is the parent body segment and the second part is the child body
segment.
Excuse me. Actually, there are three. So three hyphen separated parts.
The first one is the parent, the second part is the child,
and the third one is the axis.
Within the body segment name, once again, we have two parts separated by an underscore.
The first part indicates which broad body part it is.
So like C stands for central.
So things like the thorax, the head, as LF means left front leg.
I can read about it in more details here.
Okay, so now that we have selected 42 actuated degrees of freedom and stored them in this variable,
we can actually add them to our fly model by calling fly actuators.
In this case, we want to add position actuators, which would allow us to,
like in this demo,
set target positions
and the physics simulator is simply going to apply an appropriate amount of force
subjected to those physical parameters that would bring the model closer to the desired state.
We specify the gain. In this case, we use 50.
Optionally, we can add sites to anatomical joints, which would make it easier to visualize where
different body segments are.
I'll simply run it and show you the results later.
More importantly, we can add visuals to the fly.
So far, the fly doesn't have any color, any texture.
So we can call fly colorized simply to get a kind of visualization that looks more
like this and add a camera.
In this case, we add a tracking camera.
More information is described here, but in brief, the camera maintains an offset from the fly,
but it looks at the fly from the same global angle.
So far, we have not run any physics simulation. We have only composed a fly with
a camera attached to it.
There's no simulation going on. There's no movement.
And as I said, this is what is called a model in MuJoCo.
We can export this model. We can save it.
If we look at the file system,
you'll see under demo output, a fly model folder
where you have a bunch of STL files.
Those STL files, they are essentially shapes.
And if we go down a bit, we'll see a XML file here, neuromechfly.xml.
For some reason, it's using TextEdit to open it, but that's fine.
And in this XML file, we'll see a serialized or written description of
exactly what we have added.
If you want to go into more information on what these are, you can follow a
separate tutorial called
1B Advanced Model Composition. There's a more detailed description.
But for now, what matters is that we have composed a model. We have saved it.
Maybe you'll see in some other physics simulation software that often you start from this folder
with an XML file and a bunch of mesh files.
This is also what we did until FlyGym 2.0.0.
In our experience, we found that this way of procedurally composing a model is a little
bit easier to version control
because what happens often with these saved models is that one person passes to their collaborator
the XML file that I just showed you,
and it's just not very readable. You don't know what the previous person changed, maybe by hand,
and it just becomes confusing over time.
Again, we can see some mesh files being generated and this XML file.
Again, what we have done so far, the output of it is simply this XML file.
Everything that we have done so far is basically a glorified way of
generating this text file by code.
What we need to do next is to actually compile the model to a format that
can be actually simulated.
To do so, we call fly.compile, which generates two things,
a mj_model and a mj_data.
mj_model is simply a compiled...
or a data structure that contains information on the information that we have included,
things like which joints, which degrees of freedom there are.
The MuJoCo data right now is an empty container that can contain dynamic data such as
acceleration and forces and positions as we actually simulate the model.
Using this helper function, we can run a brief simulation using mj_model and mj_data.
The detail of it is not so important for this tutorial.
It's simply a way of showing what we have composed.
If we put what we have composed so far into an actual simulation, we can
see that the fly initializes to a neutral pose,
but because of passive, finite stiffness of non-actuated joints, the abdomen is going to settle
into a pose that is subjected to gravity.
Compared to the demonstration shown here, you'll note that this fly is standing on some sort
of ground, because if we move the legs, it's going to apply certain forces on the
ground, whereas here it's just a fly.
That's because we have so far only composed the fly and not the world that it has seen.
So in the next section, we will add a world. The most basic example is a
flat ground world where we only have a single line of code.
If you go to the API reference for NeuroMechFly, you'll see that there are
different types of worlds.
For example, we have the flat ground world, but we also have a tethered world and
also complex terrain world,
for example, environments with gaps in the ground.
These are a little bit more advanced.
You can see what these worlds look like in future tutorials,
for example, this one, for the purpose of this tutorial that we are not going to
look too much into it.
Anyway, so far we have composed the fly, we have composed the world,
we can attach the fly to the world.
Before we do that, we need to tell the model where you want the fly to
be placed in the world.
So we define a spawn position in millimeters, so 0.7 mm in height.
So that's our origin on the XY plane, but [0.7] mm above the ground.
And we're also going to define a rotation, in this case, just no rotation, a unit rotation,
specified in quaternion format. I'm not going to go into details for this, you can read
more about it here.
And once we have defined those two variables, we also need to define what kind of
contact we want between the fly and the ground.
Again, we have provided under flygym anatomy presets for which body segments should be able to
collide with the ground.
The reason why we don't want every body segment to collide with the ground is that
it's computationally expensive.
For every pair of body segments or just objects that may collide, it introduces a certain
computational cost.
So if you're simply simulating walking, there's no need, for example, to simulate to compute for
collisions between the head and the ground,
or the head and the abdomen, or between different abdominal segments,
which is why, by the way, if we reduce the stiffness, we can see the abdomen
kind of running into the thorax, which is not supposed to happen.
Anyway, what we are doing here is to filter out the collision pairs
that we don't care about.
I'm just going to run these two cells.
And once we have defined the spawn position and spawn orientation, and also which body segments
should be in contact with the ground, we can add the fly to the world.
Actually, I was wrong in this preset. We do simulate the collision between
the head and the ground.
I think the reason for this is that we wanted to explore failure modes where the
fly entirely just flips over,
and then we don't want the collision between the major body segments and the ground becomes
interesting or important.
Like before, once we have attached a fly to the world,
we can save the model, this time the world model, instead of a standalone fly.
Once again, we can see the XML file. Its content looks like this.
Once again, we can compile the model, this time compiling the world instead of just the fly,
and preview the compiled model of the world with the fly in it
in a very short simulation.
As I demonstrated actually in a previous tutorial, you can run this command locally if you
have downloaded or installed FlyGym locally to see essentially what we have here.
So one thing to note is that in the in the tutorial that I just went
through, it's not possible to do everything that you might want to do with the simulated model.
In fact, that is by design, we wanted to build a
higher level API for procedurally composing a fly.
And by design, we didn't implement every function that would tune every parameter that you can
in a MuJoCo physics simulator.
If you wish to do that, please follow tutorial 1B. I'm not going to actually go
through it in Google Colab.
But essentially, this tutorial tells you how you might want to do things like, for example,
settings, joint stiffnesses differently for certain joints, but not every joint
and also things like modifying physics parameters.
In this case, if we massively reduce the frequency of the simulation or increase the time
step of the simulation, then the physics simulation becomes very unstable.
It also tells you how to add debug things, for example, visualization of certain joints.
I encourage you to look into this tutorial if you want to do anything that it's
not covered in the first notebook.
Additionally, the API reference section of the NeuroMechFly documentation gives you a very complete explanation
of exactly what you can do and what functions are exposed and what options are.
And that's it. In the next tutorials, we'll actually cover how to run the simulation.