Simulating embodied sensorimotor control with NeuroMechFly v2¶

Note

API changes may occur in future releases. See the changelog for details.

FlyGym is the Python library for NeuroMechFly v2, a digital twin of the adult fruit fly Drosophila melanogaster that can see, smell, walk over challenging terrain, and interact with the environment (see our NeuroMechFly v2 paper).

FlyGym consists of the following components:

Biomechanical model: The biomechanical model is based on a micro-CT scan of a real adult female fly (see our original NeuroMechFly publication). In NeuroMechFly v2 we have updated several body segments (in particular in the antennae) to better reflect the biological reality.
Vision: The fly has compound eyes consisting of individual units called ommatidia. These are arranged on a hexagonal lattice. We have simulated visual inputs on the fly’s retinas and corresponding visual neuron activities using the FlyVision model (see Lappalainen et al., Nature, 2024).
Olfaction: The fly has odor receptors in the antennae and maxillary palps. We have simulated olfactory signals experienced by the fly by computing odor/chemical intensities at the locations of the antennae and maxillary palps.
Hierarchical control: The fly’s central nervous system (CNS) consists of the brain and the ventral nerve cord (VNC). This hierarchy is analogous to the brain-spinal cord organization of vertebrates. The user can build a two-part model — one handling brain-level sensory integration and decision making as well as one handling VNC-level motor control — with a interface between the two consisting of descending (brain-to-VNC) and ascending (VNC-to-brain) neuronal pathways.
Leg adhesion: Insects have evolved specialized adhesive structures at the tips of their legs. These enable locomotion on inclined and overhanging surfaces. We have simulated adhesion in our model. The mechanism by which the fly lifts the legs during locomotion despite adhesive forces is currently not well understood. Therefore, to abstract this, adhesion can be turned on or off during stance or swing of the legs, respectively.
Mechanosensory feedback: The user has access to joint angles, joint torques, and collision/contact forces experienced by the simulated fly.

FlyGym formulates the control of the simulated fly as a partially observable Markov Decision Process (MDP) and implements the Gym interface. This allows the user to use a wide range of reinforcement learning (RL) algorithms to train the fly to perform tasks. The standardized interface also allows the user to easily implement their own premotor computation and/or sensory preprocessing.

The biomechanical model and its interaction with the environment are encapsulated as a MDP task. A user-defined controller interfaces with the task through actions (red) and observations (blue). The user can extend the MDP task by adding preprogrammed processing routines for sensory inputs (purple) and motor outputs (light blue), to modify the action and observation spaces handled by the controller.¶

Citation¶

If you use FlyGym or NeuroMechFly in your research, please cite the following two papers:

% Original NeuroMechFly: Original biomechanical model, kinematic replay, CPG-based
% neural controller, PyBullet version, DoF analysis, perturbation tests
@article{LobatoRios2022,
   doi = {10.1038/s41592-022-01466-7},
   url = {https://doi.org/10.1038/s41592-022-01466-7},
   year = {2022},
   month = may,
   volume = {19},
   number = {5},
   pages = {620--627},
   author = {Victor Lobato-Rios and Shravan Tata Ramalingasetty and Pembe Gizem \"{O}zdil and Jonathan Arreguit and Auke Jan Ijspeert and Pavan Ramdya},
   title = {{NeuroMechFly}, a neuromechanical model of adult {Drosophila} melanogaster},
   journal = {Nature Methods}
}

% NeuroMechFly v2: This library, MuJoCo version, leg adhesion, rule-based controller,
% hybrid controller, complex terrain, preprogrammed steps, leg adhesion, vision,
% olfaction, RL-based navigation, Gym environment, updated biomechanical model
@article{WangChen2023,
   author = {Sibo Wang-Chen and Victor Alfred Stimpfling and Pembe Gizem \"{O}zdil and Louise Genoud and Femke Hurtak and Pavan Ramdya},
   title = {NeuroMechFly 2.0, a framework for simulating embodied sensorimotor control in adult Drosophila},
   year = {2023},
   doi = {10.1101/2023.09.18.556649},
   URL = {https://www.biorxiv.org/content/10.1101/2023.09.18.556649},
   journal = {bioRxiv}
}

Note

Privacy policy: This site uses Google Analytics to collect data about your interactions with our website. This includes information such as your IP address, browsing behavior, and device type. We use this data to improve our website and understand user preferences. Google Analytics uses Cookies, which are small text files stored on your device. See How Google uses information from sites or apps that use our services. To opt-out, you can use a browser extension to deactivate Google Analytics.