WEBVTT 00:00:01.080 --> 00:00:08.280 Welcome to the third NeuroMechFly video tutorial. In this tutorial, we'll cover how to compose 00:00:08.280 --> 00:00:15.140 a simulation with a fly and maybe a fly in a certain environment using the 00:00:15.140 --> 00:00:17.000 Python interface for NeuroMechFly. 00:00:18.060 --> 00:00:27.074 Specifically, we're going to follow the code tutorial notebooks available on the website here. 00:00:27.074 --> 00:00:33.940 Again, the current version of FlyGym is 2.1.0. Future versions might deviate from this video 00:00:33.940 --> 00:00:43.600 tutorial, so please always follow the tutorials on the webpage instead of following what I'm doing 00:00:43.600 --> 00:00:45.480 verbatim in this video. 00:00:47.240 --> 00:00:53.040 We have already covered four different ways to install FlyGym in the previous tutorial. Since we 00:00:53.040 --> 00:00:57.580 don't have a lot of computation going on in just composing the model, and we don't 00:00:57.580 --> 00:01:00.700 really care about performance, I'm just going to use Google Colab. 00:01:02.760 --> 00:01:08.040 I have already run the cell that installs FlyGym. 00:01:09.620 --> 00:01:16.720 You may need to start by connecting to an instance and clicking this button, 00:01:16.720 --> 00:01:18.040 and it might take some time. 00:01:18.040 --> 00:01:33.620 So, composing models and scenes. Basically, models in MuJoCo, the underlying physics simulation for FlyGym, has 00:01:33.620 --> 00:01:40.280 a format called MJCF, MuJoCo Modeling Format, that defines what's being simulated. 00:01:40.280 --> 00:01:46.600 Maybe we should clarify that the word model is a little bit ambiguous. We kind of 00:01:46.600 --> 00:01:50.271 use it interchangeably with simulation, 00:01:50.271 --> 00:01:59.080 but in the context of MuJoCo, MuJoCo model refers to literally the 00:01:59.080 --> 00:02:05.020 set of things that have been simulated and their physical properties like surface friction. 00:02:05.020 --> 00:02:16.220 It does not contain dynamical information like forces and positions that you would read out when 00:02:16.220 --> 00:02:17.880 you actually run the simulation. 00:02:20.620 --> 00:02:30.460 The composition workflow of FlyGym is a high level wrapper around MJCF. 00:02:33.780 --> 00:02:41.920 And it is exposed through FlyGym.compose. In this case, we start by initializing a fly. 00:02:42.220 --> 00:02:50.320 So we create an empty NeuroMechFly model, very straightforward. You can give it a name 00:02:50.320 --> 00:02:52.040 by default as NeuroMechFly. 00:02:54.940 --> 00:03:00.420 And now we have an empty fly model with nothing attached to it. So next we 00:03:00.420 --> 00:03:06.220 want to add some joints to it. And to add the joints, we need to have 00:03:06.220 --> 00:03:08.720 two pieces of information. 00:03:09.460 --> 00:03:17.060 First, we need to know what joints there are. This involves both knowing which anatomical joints 00:03:17.060 --> 00:03:23.420 there are. For example, is there a joint between those two leg segments? Is there a 00:03:23.420 --> 00:03:29.980 joint between the tibia and tarsus of a leg, 00:03:29.980 --> 00:03:37.400 but also joint degrees of freedom? So for example, is there is the joint between tibia 00:03:37.400 --> 00:03:44.380 and tarsus a ball joint where you can rotate in all three about all three axes? 00:03:44.600 --> 00:03:49.360 We know this is not true. In fact, it's a one degree of freedom joint where 00:03:49.360 --> 00:03:53.580 it acts more like a hinge instead of a ball joint. 00:03:53.580 --> 00:03:59.280 So this kind of information needs to be defined. Another thing that needs to be defined 00:03:59.280 --> 00:04:06.940 is the convention on the ordering of different rotational axes. 00:04:06.940 --> 00:04:14.600 If you want to go deeper into this mathematically, the reason why this is important is 00:04:14.600 --> 00:04:27.360 that 3D rotations, as you can see on this Wikipedia page, 3D rotations do not commute. 00:04:28.320 --> 00:04:31.460 So the ordering of them is actually important. 00:04:34.220 --> 00:04:42.840 Additionally, once we have decided which joints we want to add, we also want to add 00:04:42.840 --> 00:04:50.240 joint articulation, meaning that we find a subset of joints that we want to actively control 00:04:50.240 --> 00:04:58.360 as in actively apply forces to instead of using only their passive properties like stiffness. 00:04:59.460 --> 00:05:06.380 We also want to define a neutral pose for the fly. So in order to do 00:05:06.380 --> 00:05:12.360 all of this, we want to implement a couple of things from flygym.anatomy. 00:05:14.440 --> 00:05:19.680 We have a lot of presets. So for most simulations, you can simply say I want 00:05:19.680 --> 00:05:25.820 the axis order to be roll first, pitch next, and yaw last. We want the joints 00:05:25.820 --> 00:05:33.520 to be the set of all biological joints. Biological meaning that, for example, if we look 00:05:33.520 --> 00:05:39.420 at the tibia-tarsus joint, it is physically possible to have three rotational degrees of freedom, 00:05:40.000 --> 00:05:42.340 whereas biologically, the biological 00:05:42.340 --> 00:05:48.460 reality is that there's only one. From there, we can set up a skeleton 00:05:48.460 --> 00:05:55.180 of the fly. So we want to use the preset of joints, all biological joints, and 00:05:55.180 --> 00:05:57.880 this particular axis order. 00:06:01.360 --> 00:06:08.100 Next, we can set up a neutral pose. We already have some presets. But you can 00:06:08.100 --> 00:06:11.100 also supply a dictionary of different joint angles. 00:06:15.680 --> 00:06:27.920 Once we have the skeleton and the neutral pose set up, we can basically add them 00:06:27.920 --> 00:06:32.200 to the fly. I'll skip the cell block for now because it's not so crucial. 00:06:36.320 --> 00:06:44.280 To add the joints, we use the fly.add_joint method with a specification of the skeleton 00:06:44.280 --> 00:06:46.220 as well as the neutral pose. 00:06:51.780 --> 00:06:57.560 The cell block basically goes back to my point that 3D rotations do not commute. 00:06:57.560 --> 00:06:58.420 The ordering is important. 00:06:59.400 --> 00:07:03.397 So if we want to get the same, if we want to 00:07:03.397 --> 00:07:07.060 express the same neutral pose in roll yaw pitch versus pitch 00:07:07.060 --> 00:07:10.500 roll yaw ordering, we actually get different numbers. 00:07:14.220 --> 00:07:25.760 Now we have created an empty fly and added joints to it. The next step 00:07:25.760 --> 00:07:30.360 is to add actuation to a subset of the joints. 00:07:32.640 --> 00:07:35.187 Once again, under FlyGym Anatomy, 00:07:35.187 --> 00:07:41.300 we have provided a bunch of presets of actuated degrees of freedom, 00:07:43.060 --> 00:07:49.300 namely all, which means all articulated joints, 00:07:49.300 --> 00:07:56.329 in this case all biological joints, will be actuated. Another option is legs only, 00:07:56.329 --> 00:08:09.615 so I'll activate, not activate, but actuate active actuators to all joints that are on the legs. 00:08:09.900 --> 00:08:14.189 And there's also another option called legs active only. 00:08:14.189 --> 00:08:18.340 So the distinction between legs only and legs active only 00:08:18.340 --> 00:08:31.700 is that there are further joints on the legs that are not actively actuated, biologically speaking. 00:08:32.120 --> 00:08:47.300 So if we look at the tarsus joints, so tarsus is this last segment, you'll see 00:08:47.300 --> 00:08:54.040 that there are joints between different tarsal segments, but the biological reality is that it's soft. 00:08:54.040 --> 00:09:02.720 It has several segments, but every segment is so small that they are just quite compliant. 00:09:07.140 --> 00:09:12.840 So one thing that we can do if we had selected legs only is that these 00:09:12.840 --> 00:09:18.740 joints would be actuated as well, but a better way to do it is to use 00:09:18.740 --> 00:09:26.900 legs active only, which would give you only actuators at the other joints, but not here. 00:09:27.720 --> 00:09:33.480 It's important to note that maybe the biological reality is not so black and white, because 00:09:33.480 --> 00:09:40.620 in fact there's a tendon, the long tendon, that goes from the tip of the tarsus 00:09:40.620 --> 00:09:42.680 all the way up. 00:09:43.640 --> 00:09:48.880 So in a way there is active actuation, but the actuation of this joint and these 00:09:48.880 --> 00:09:51.080 joints, it's just a little bit complicated. 00:09:53.180 --> 00:09:57.100 Anyway, in this case we choose legs active only. 00:10:00.420 --> 00:10:03.840 The actuated degrees of freedom preset is a filter. 00:10:04.280 --> 00:10:11.162 It depends on what joints exist in the first place when we added the joints. 00:10:11.678 --> 00:10:18.483 So we actually have to call this get_actuated_dofs[_from_preset] method 00:10:18.483 --> 00:10:20.488 on the skeleton object. 00:10:20.488 --> 00:10:25.960 So if we run it, we say, okay, previously we have added all biological joints to 00:10:25.960 --> 00:10:35.400 the skeleton and among these joints we want to actuate all joints that are on the 00:10:35.400 --> 00:10:37.080 legs that are supposed to be active. 00:10:37.080 --> 00:10:47.020 And it's going to tell us that we have 42 actuated degrees of freedom, which is 00:10:47.020 --> 00:10:50.420 correct because the fly has seven of them per leg and six legs. 00:10:51.220 --> 00:10:54.160 We can even run this to see a full list of them. 00:11:04.620 --> 00:11:10.560 If we just look at the left front legs, those are the seven actuated joints or 00:11:10.560 --> 00:11:12.040 degrees of freedom. 00:11:15.480 --> 00:11:18.380 Body segments and joints and everything are semantically named. 00:11:21.360 --> 00:11:28.680 So the joints, for example, have their names have two parts separated by a hyphen. 00:11:28.840 --> 00:11:33.680 The first part is the parent body segment and the second part is the child body 00:11:33.680 --> 00:11:34.180 segment. 00:11:34.560 --> 00:11:40.040 Excuse me. Actually, there are three. So three hyphen separated parts. 00:11:40.180 --> 00:11:43.020 The first one is the parent, the second part is the child, 00:11:43.020 --> 00:11:46.160 and the third one is the axis. 00:11:48.160 --> 00:11:54.720 Within the body segment name, once again, we have two parts separated by an underscore. 00:11:55.260 --> 00:12:01.420 The first part indicates which broad body part it is. 00:12:01.740 --> 00:12:04.200 So like C stands for central. 00:12:04.420 --> 00:12:10.160 So things like the thorax, the head, as LF means left front leg. 00:12:10.160 --> 00:12:13.960 I can read about it in more details here. 00:12:15.520 --> 00:12:23.060 Okay, so now that we have selected 42 actuated degrees of freedom and stored them in this variable, 00:12:23.060 --> 00:12:28.880 we can actually add them to our fly model by calling fly actuators. 00:12:29.240 --> 00:12:34.034 In this case, we want to add position actuators, which would allow us to, 00:12:35.096 --> 00:12:37.080 like in this demo, 00:12:43.076 --> 00:12:44.700 set target positions 00:12:46.420 --> 00:12:50.880 and the physics simulator is simply going to apply an appropriate amount of force 00:12:50.880 --> 00:12:56.860 subjected to those physical parameters that would bring the model closer to the desired state. 00:12:58.560 --> 00:13:02.860 We specify the gain. In this case, we use 50. 00:13:08.840 --> 00:13:15.600 Optionally, we can add sites to anatomical joints, which would make it easier to visualize where 00:13:15.600 --> 00:13:16.940 different body segments are. 00:13:17.380 --> 00:13:20.120 I'll simply run it and show you the results later. 00:13:22.560 --> 00:13:26.060 More importantly, we can add visuals to the fly. 00:13:27.360 --> 00:13:30.960 So far, the fly doesn't have any color, any texture. 00:13:31.380 --> 00:13:36.260 So we can call fly colorized simply to get a kind of visualization that looks more 00:13:36.260 --> 00:13:38.460 like this and add a camera. 00:13:38.660 --> 00:13:40.640 In this case, we add a tracking camera. 00:13:42.000 --> 00:13:50.797 More information is described here, but in brief, the camera maintains an offset from the fly, 00:13:51.660 --> 00:13:58.300 but it looks at the fly from the same global angle. 00:14:00.660 --> 00:14:07.900 So far, we have not run any physics simulation. We have only composed a fly with 00:14:07.900 --> 00:14:09.260 a camera attached to it. 00:14:09.260 --> 00:14:12.340 There's no simulation going on. There's no movement. 00:14:13.080 --> 00:14:17.820 And as I said, this is what is called a model in MuJoCo. 00:14:18.540 --> 00:14:21.500 We can export this model. We can save it. 00:14:23.760 --> 00:14:25.740 If we look at the file system, 00:14:30.040 --> 00:14:34.500 you'll see under demo output, a fly model folder 00:14:35.900 --> 00:14:41.140 where you have a bunch of STL files. 00:14:41.300 --> 00:14:46.740 Those STL files, they are essentially shapes. 00:14:52.160 --> 00:14:59.460 And if we go down a bit, we'll see a XML file here, neuromechfly.xml. 00:15:11.180 --> 00:15:16.240 For some reason, it's using TextEdit to open it, but that's fine. 00:15:17.480 --> 00:15:25.000 And in this XML file, we'll see a serialized or written description of 00:15:25.000 --> 00:15:26.460 exactly what we have added. 00:15:28.460 --> 00:15:37.160 If you want to go into more information on what these are, you can follow a 00:15:37.160 --> 00:15:38.600 separate tutorial called 00:15:41.380 --> 00:15:45.940 1B Advanced Model Composition. There's a more detailed description. 00:15:46.040 --> 00:15:50.640 But for now, what matters is that we have composed a model. We have saved it. 00:15:52.080 --> 00:16:00.460 Maybe you'll see in some other physics simulation software that often you start from this folder 00:16:00.460 --> 00:16:04.280 with an XML file and a bunch of mesh files. 00:16:04.880 --> 00:16:09.840 This is also what we did until FlyGym 2.0.0. 00:16:11.040 --> 00:16:17.820 In our experience, we found that this way of procedurally composing a model is a little 00:16:17.820 --> 00:16:20.460 bit easier to version control 00:16:20.460 --> 00:16:27.280 because what happens often with these saved models is that one person passes to their collaborator 00:16:27.280 --> 00:16:29.900 the XML file that I just showed you, 00:16:30.080 --> 00:16:35.720 and it's just not very readable. You don't know what the previous person changed, maybe by hand, 00:16:35.720 --> 00:16:38.460 and it just becomes confusing over time. 00:16:42.960 --> 00:16:47.380 Again, we can see some mesh files being generated and this XML file. 00:16:48.980 --> 00:16:55.860 Again, what we have done so far, the output of it is simply this XML file. 00:16:57.300 --> 00:17:02.069 Everything that we have done so far is basically a glorified way of 00:17:02.069 --> 00:17:06.460 generating this text file by code. 00:17:07.740 --> 00:17:13.060 What we need to do next is to actually compile the model to a format that 00:17:13.060 --> 00:17:15.880 can be actually simulated. 00:17:15.880 --> 00:17:23.160 To do so, we call fly.compile, which generates two things, 00:17:23.160 --> 00:17:24.180 a mj_model and a mj_data. 00:17:25.280 --> 00:17:28.536 mj_model is simply a compiled... 00:17:34.006 --> 00:17:39.201 or a data structure that contains information on the information that we have included, 00:17:39.520 --> 00:17:42.220 things like which joints, which degrees of freedom there are. 00:17:42.220 --> 00:17:48.940 The MuJoCo data right now is an empty container that can contain dynamic data such as 00:17:48.940 --> 00:17:55.460 acceleration and forces and positions as we actually simulate the model. 00:17:58.320 --> 00:18:05.940 Using this helper function, we can run a brief simulation using mj_model and mj_data. 00:18:07.420 --> 00:18:11.960 The detail of it is not so important for this tutorial. 00:18:13.300 --> 00:18:16.000 It's simply a way of showing what we have composed. 00:18:19.360 --> 00:18:24.140 If we put what we have composed so far into an actual simulation, we can 00:18:24.140 --> 00:18:27.480 see that the fly initializes to a neutral pose, 00:18:27.480 --> 00:18:38.600 but because of passive, finite stiffness of non-actuated joints, the abdomen is going to settle 00:18:38.600 --> 00:18:44.860 into a pose that is subjected to gravity. 00:18:49.280 --> 00:18:56.500 Compared to the demonstration shown here, you'll note that this fly is standing on some sort 00:18:56.500 --> 00:19:03.180 of ground, because if we move the legs, it's going to apply certain forces on the 00:19:03.180 --> 00:19:05.080 ground, whereas here it's just a fly. 00:19:05.640 --> 00:19:11.549 That's because we have so far only composed the fly and not the world that it has seen. 00:19:12.300 --> 00:19:18.020 So in the next section, we will add a world. The most basic example is a 00:19:18.020 --> 00:19:25.620 flat ground world where we only have a single line of code. 00:19:26.740 --> 00:19:33.140 If you go to the API reference for NeuroMechFly, you'll see that there are 00:19:33.140 --> 00:19:35.740 different types of worlds. 00:19:35.740 --> 00:19:42.060 For example, we have the flat ground world, but we also have a tethered world and 00:19:42.060 --> 00:19:43.660 also complex terrain world, 00:19:44.000 --> 00:19:52.600 for example, environments with gaps in the ground. 00:19:54.220 --> 00:19:55.800 These are a little bit more advanced. 00:19:55.800 --> 00:20:09.080 You can see what these worlds look like in future tutorials, 00:20:10.200 --> 00:20:16.580 for example, this one, for the purpose of this tutorial that we are not going to 00:20:16.580 --> 00:20:17.800 look too much into it. 00:20:19.260 --> 00:20:22.691 Anyway, so far we have composed the fly, we have composed the world, 00:20:22.691 --> 00:20:25.200 we can attach the fly to the world. 00:20:26.120 --> 00:20:30.300 Before we do that, we need to tell the model where you want the fly to 00:20:30.300 --> 00:20:31.320 be placed in the world. 00:20:31.680 --> 00:20:40.760 So we define a spawn position in millimeters, so 0.7 mm in height. 00:20:42.260 --> 00:20:47.920 So that's our origin on the XY plane, but [0.7] mm above the ground. 00:20:48.680 --> 00:20:58.217 And we're also going to define a rotation, in this case, just no rotation, a unit rotation, 00:20:58.800 --> 00:21:04.400 specified in quaternion format. I'm not going to go into details for this, you can read 00:21:04.400 --> 00:21:05.800 more about it here. 00:21:06.740 --> 00:21:12.660 And once we have defined those two variables, we also need to define what kind of 00:21:12.660 --> 00:21:15.660 contact we want between the fly and the ground. 00:21:16.640 --> 00:21:25.660 Again, we have provided under flygym anatomy presets for which body segments should be able to 00:21:25.660 --> 00:21:27.680 collide with the ground. 00:21:28.680 --> 00:21:34.780 The reason why we don't want every body segment to collide with the ground is that 00:21:34.780 --> 00:21:36.800 it's computationally expensive. 00:21:37.180 --> 00:21:42.920 For every pair of body segments or just objects that may collide, it introduces a certain 00:21:42.920 --> 00:21:43.960 computational cost. 00:21:44.540 --> 00:21:51.720 So if you're simply simulating walking, there's no need, for example, to simulate to compute for 00:21:51.720 --> 00:21:53.720 collisions between the head and the ground, 00:21:53.720 --> 00:22:00.060 or the head and the abdomen, or between different abdominal segments, 00:22:01.980 --> 00:22:08.080 which is why, by the way, if we reduce the stiffness, we can see the abdomen 00:22:08.080 --> 00:22:11.980 kind of running into the thorax, which is not supposed to happen. 00:22:13.840 --> 00:22:18.828 Anyway, what we are doing here is to filter out the collision pairs 00:22:18.828 --> 00:22:20.140 that we don't care about. 00:22:21.240 --> 00:22:26.140 I'm just going to run these two cells. 00:22:27.980 --> 00:22:35.460 And once we have defined the spawn position and spawn orientation, and also which body segments 00:22:35.460 --> 00:22:41.360 should be in contact with the ground, we can add the fly to the world. 00:22:42.560 --> 00:22:46.471 Actually, I was wrong in this preset. We do simulate the collision between 00:22:46.471 --> 00:22:47.860 the head and the ground. 00:22:48.320 --> 00:22:55.220 I think the reason for this is that we wanted to explore failure modes where the 00:22:55.220 --> 00:22:57.764 fly entirely just flips over, 00:22:58.360 --> 00:23:05.280 and then we don't want the collision between the major body segments and the ground becomes 00:23:05.280 --> 00:23:07.140 interesting or important. 00:23:08.120 --> 00:23:12.160 Like before, once we have attached a fly to the world, 00:23:12.740 --> 00:23:16.960 we can save the model, this time the world model, instead of a standalone fly. 00:23:17.860 --> 00:23:22.100 Once again, we can see the XML file. Its content looks like this. 00:23:22.580 --> 00:23:28.785 Once again, we can compile the model, this time compiling the world instead of just the fly, 00:23:29.960 --> 00:23:35.791 and preview the compiled model of the world with the fly in it 00:23:35.791 --> 00:23:37.980 in a very short simulation. 00:23:41.020 --> 00:23:46.520 As I demonstrated actually in a previous tutorial, you can run this command locally if you 00:23:46.520 --> 00:23:52.920 have downloaded or installed FlyGym locally to see essentially what we have here. 00:24:02.040 --> 00:24:06.380 So one thing to note is that in the in the tutorial that I just went 00:24:06.380 --> 00:24:15.398 through, it's not possible to do everything that you might want to do with the simulated model. 00:24:16.260 --> 00:24:20.000 In fact, that is by design, we wanted to build a 00:24:20.000 --> 00:24:27.020 higher level API for procedurally composing a fly. 00:24:27.020 --> 00:24:34.320 And by design, we didn't implement every function that would tune every parameter that you can 00:24:34.320 --> 00:24:35.860 in a MuJoCo physics simulator. 00:24:36.600 --> 00:24:41.140 If you wish to do that, please follow tutorial 1B. I'm not going to actually go 00:24:41.140 --> 00:24:43.080 through it in Google Colab. 00:24:43.080 --> 00:24:51.480 But essentially, this tutorial tells you how you might want to do things like, for example, 00:24:51.960 --> 00:24:58.160 settings, joint stiffnesses differently for certain joints, but not every joint 00:24:58.680 --> 00:25:02.200 and also things like modifying physics parameters. 00:25:03.080 --> 00:25:10.260 In this case, if we massively reduce the frequency of the simulation or increase the time 00:25:10.260 --> 00:25:14.800 step of the simulation, then the physics simulation becomes very unstable. 00:25:16.900 --> 00:25:22.060 It also tells you how to add debug things, for example, visualization of certain joints. 00:25:22.060 --> 00:25:32.320 I encourage you to look into this tutorial if you want to do anything that it's 00:25:32.320 --> 00:25:34.600 not covered in the first notebook. 00:25:35.460 --> 00:25:43.260 Additionally, the API reference section of the NeuroMechFly documentation gives you a very complete explanation 00:25:43.260 --> 00:25:51.900 of exactly what you can do and what functions are exposed and what options are. 00:25:53.200 --> 00:25:59.080 And that's it. In the next tutorials, we'll actually cover how to run the simulation.