So, the goals for today are really didactic,
so the idea is to try to give you a background of NeuroMechFly as a
model, show you where you
can find these resources, and incidentally, all of this, including this video, should
be posted on our website after the workshop,
so you can return back to it, including the slides, give you an overview of how
we've actually used NeuroMechFly most recently,
and then in blue we have the actual on-the-ground work, installation, and helping you
to perform your own simulations.
So, the last part of this workshop is going to be you just trying to implement
things you find interesting.
Okay, so why NeuroMechFly?
This is the sort of historical aspect of it.
So, I'm not going to bring up Feynman, even though I just brought up Feynman,
but there's this idea of reverse engineering,
or what I cannot create, I do not understand,
and the idea of reverse engineering is to disassemble or analyze something in detail,
and as a biologist, we're all reverse engineering in the sense that
we're disassembling or analyzing something in detail.
Ideally, we discover important concepts regarding those details that we can translate to robotics,
artificial systems, or to humans,
but the neat thing about what we're doing with NeuroMechFly and I think other body models
is to produce something similar, right,
and we have several people here from laboratories that are trying to build Drosophila-inspired
robots and people working on simulations,
and I think this is really one of the impetus for those works is to try
to produce something similar, to have impact in the real world.
And so, we're not the first people to generate fly or insect-inspired systems,
so this is an old video from Quinn's Whegs II robot, where you have these hybrid Whegs,
which are wheel or leg-like structures that allow this insect robot to travel over complex terrain.
Of course, it looks nothing like the beautiful movements of a cockroach leg.
As well, we've had simulations, 3D physics simulations of insect behavior,
so this is for Ansgar Büschges's lab from 2004, where they built a
model of the stick insect
and have controllers based on what they understand about the underlying biology of the stick
insect to drive movements of this controller.
So this is not new, and indeed, we also developed
a much simpler model of Drosophila in 2017 that I'm not going to show you,
but of course, the question is why work with Drosophila instead of one of these other insects, right?
We looked at the cockroach,
they have this beautiful behavior where they can go over complex terrain,
stick insect as well can move on stems in nature,
and of course, I think everybody in this room is aware that flies generate complex behaviors
and importantly, we can actually target neurons of interest.
We have a small compact nervous system,
and these are really the properties that allow us to dig really deeply into the underlying
mechanisms of control in the fly.
So the first resource that I would refer you to
is the first original NeuroMechFly paper in 2022.
This is the work that first defined our body model, led by Victor Lobato-Rios,
a PhD student in my laboratory,
and what that entailed was taking a real fly, embedding it here,
and then performing a CT scan of the fly.
So at this point, we are really interested in moving beyond ball and stick models to
much more complex models in terms of morphological realism,
and part of the reason is that we eventually want to actually make inferences about what
the real animal experience is in terms of the mechanical feedback with the environment,
as well as, for example, if the fly is grooming, as you would be interested in the Simpson lab,
we need to have real potential interactions between the body parts during collisions between
the antenna and the legs, for example.
You can't do this as well with ball and stick models, which motivated us
to buil this initial body model.
So using this body model, we could actually do some really nice things in closed loop
with real data.
So here we have a fly on the ball,
so we can make some inferences about the mechanical sensory feedback of this animal.
So here we have 2D tracking, then 3D reconstruction, what we call kinematic replay, where we replay
the behavior that we measured in a real fly, and then we can infer the collisions
that occur between the different body parts of the animal.
So this is one really nice early way in which the biomechanical model
could be useful for science.
But of course, if we want to understand how the nervous system works to drive behavior,
we need something that can actually have neural network controllers as well,
which leads us to the paper that is now in press in Nature Methods and was
published in bioRxiv, another resource where we describe basically what I'm just going to very briefly
overview in the following slides of the improvements on the initial model.
This is kind of busy, but effectively what you can see that we've added vision, olfaction,
we've added as well the opportunity of using descending and ascending signals, we've added
leg adhesion, the possibility for rugged terrain as you see here in our new model, and
that allowed us to do a variety of different things in terms of exploring
mechanisms for neural control.
Well, the first thing we wanted to do actually is to try to generate more realistic locomotion.
So initially we had our locomotor controller driven by CPG controllers to try to emulate
basically the joint degree of freedom ranges we saw in the real fly.
Here what we did is we took a single step from a locomoting fly,
performed 3D pose estimation,
and then used this information to actually drive locomotion in the real fly.
So the initial use cases were locomotor.
So the first thing is by adding adhesion we can actually have our simulation walking on
increasingly slanted surfaces.
Another use case that I'll just briefly talk about is that we have CPG based locomotor
control, or we can have feedback based control, so the sort of dichotomy between distributed
control versus centrally generated control with CPGs.
Another use case was to add vision and olfaction.
So in this visual object following use case you can actually see what the simulation observes.
So here we have the left eye and the right eye, the set of ommatidia, and
you can actually watch what the fly is actually observing.
So here we have this ball that it's following, and we're using a controller
to actually follow the ball.
In the olfactory avoidance and attraction we have the fly walking through an environment
and detecting this aversive odorant and being attracted to this attractive odorant.
We also incorporated head stabilization where we can actually, so if we observe our initial
model walking through even flat terrain we see that the raw visual experience is actually
quite turbulent, but we actually actuate the neck to try to reduce the turbulence in the
way that a real insect would try to do that as well.
We also use reinforcement learning to train a policy that allows our fly to try to
reach an attractive orange odorant by passing this black pillar.
You can see again visual experience of our simulation on the left and the right eye
as well as the olfactory experience as shown by the amplitude of olfactory input.
Two more cases that were more biologically realistic per se.
This is actually an algorithm that's derived from Demir et al.
eLife where we have our fly walking through an odor plume trying
to get to the odor source.
So you can see a zoom in of what the fly is doing as it's trying
to actually reach the source here.
And finally, it's great that Srini just walked in,
we take the Lappalainen model, a visual system and use it for our fly to actually
track the other fly through a rugged terrain.
So here we can see the left eye input, right eye input and the activity patterns
across an array of different sets of neurons.
So additional resources that you could turn to, and again remember that the slides will
be available for you on our website, you
just go to neuromechfly.org.
One of the really valuable things that I think if you want to follow up on some of the work
that we've already done is these tutorials which follow through each of the points that
I just showed you.
Here are some examples.
Interacting with NeuroMechFly, controlling with CPGs, rule-based controller etc.,
vision, olfaction, path integration.
So the hope is that after today, after installing thing s and trying them out, you can yourself
just go home and try some of these tutorials yourself if you didn't manage to have
the time currently.