⚠️ Running on Google Colab? Execution there can be up to ~50× slower than on a local machine or dedicated GPU, depending on the resources Colab makes available at the time. Use Colab only for testing and following along with the tutorials—not for production runs or benchmarking.

⚠️ FlyBody integration is experimental: The API may change in future releases.

In [1]:

colab-setup

# --- Google Colab setup ---
# If you opened this notebook in Google Colab, run this cell first to
# install FlyGym and enable headless (EGL) rendering. It is a no-op when
# the notebook is run anywhere else.
import os
import sys

if "google.colab" in sys.modules:
    os.environ["MUJOCO_GL"] = "egl"
    os.environ["PYOPENGL_PLATFORM"] = "egl"
    %pip install -q tqdm "flygym @ git+https://github.com/NeLy-EPFL/flygym.git@v2.1.0"

# --- Google Colab setup --- # If you opened this notebook in Google Colab, run this cell first to # install FlyGym and enable headless (EGL) rendering. It is a no-op when # the notebook is run anywhere else. import os import sys if "google.colab" in sys.modules: os.environ["MUJOCO_GL"] = "egl" os.environ["PYOPENGL_PLATFORM"] = "egl" %pip install -q tqdm "flygym @ git+https://github.com/NeLy-EPFL/flygym.git@v2.1.0"

Replaying experimental recordings with the FlyBody model¶

In this tutorial we replay a snippet of experimentally recorded fly walking on a ball using the FlyBody body model — published in Vaxenburg et al. (2025) (code) — from the Turaga lab, while retaining the FlyGym API for scene composition, simulation, and control.

This mirrors Tutorial 2, which replays a recording with the default NeuroMechFly model, but here we drive the FlyBody skeleton and additionally demonstrate its tendon actuators. The recording is the same dataset presented in the original NeuroMechFly publication; custom code based on SeqIKPy was used to extract the kinematic chain and run the inverse kinematics. See Tutorial 5b for running a walking controller on the same model.

!!! warning "Experimental" Support for the FlyBody body model is experimental. The API may change in future releases, and not all features available for the default NeuroMechFly model are currently supported.

Load the motion snippet¶

The data is bundled as a clip under flygym_demo/ball_flybody_data/assets/ball_flybody_clip.npz. It already matches the MotionSnippet schema — the same helper used in Tutorial 2 — so we can load it directly and move on to replaying the motion.

In [2]:

from importlib.resources import files
from pathlib import Path

import numpy as np
from tqdm import trange

from flygym import Simulation
from flygym.compose import ActuatorType, FlatGroundWorld, KinematicPosePreset
from flygym.compose.fly import FlyBody
from flygym.flybody import (
    FlyBodyActuatedDOFPreset,
    FlyBodyAxisOrder,
    FlyBodyContactBodiesPreset,
    FlyBodyJointPreset,
    FlyBodySkeleton,
)
from flygym.utils.math import Rotation3D
from flygym_demo.spotlight_data import MotionSnippet

snippet_path = (
    Path(str(files("flygym_demo"))) / "ball_flybody_data/assets/ball_flybody_clip.npz"
)
snippet = MotionSnippet(snippet_path, angles_global2anatomical=False)

print("Experimental data and metadata:")
print(f"  legs: {snippet.legs}")
print(f"  dofs_per_leg: {snippet.dofs_per_leg}")
print(f"  data_fps: {snippet.data_fps}")
print(f"  experiment_trial: {snippet.experiment_trial}")
print(f"  framerange_in_raw_recording: {snippet.framerange_in_raw_recording}")
print(
    f"  joint_angles: shape={snippet.joint_angles.shape}, dtype={snippet.joint_angles.dtype}"
)
print(
    f"  fwdkin_egoxyz: shape={snippet.fwdkin_egoxyz.shape}, dtype={snippet.fwdkin_egoxyz.dtype}"
)
print(
    f"  rawpred_egoxyz: shape={snippet.rawpred_egoxyz.shape}, dtype={snippet.rawpred_egoxyz.dtype}"
)

from importlib.resources import files from pathlib import Path import numpy as np from tqdm import trange from flygym import Simulation from flygym.compose import ActuatorType, FlatGroundWorld, KinematicPosePreset from flygym.compose.fly import FlyBody from flygym.flybody import ( FlyBodyActuatedDOFPreset, FlyBodyAxisOrder, FlyBodyContactBodiesPreset, FlyBodyJointPreset, FlyBodySkeleton, ) from flygym.utils.math import Rotation3D from flygym_demo.spotlight_data import MotionSnippet snippet_path = ( Path(str(files("flygym_demo"))) / "ball_flybody_data/assets/ball_flybody_clip.npz" ) snippet = MotionSnippet(snippet_path, angles_global2anatomical=False) print("Experimental data and metadata:") print(f" legs: {snippet.legs}") print(f" dofs_per_leg: {snippet.dofs_per_leg}") print(f" data_fps: {snippet.data_fps}") print(f" experiment_trial: {snippet.experiment_trial}") print(f" framerange_in_raw_recording: {snippet.framerange_in_raw_recording}") print( f" joint_angles: shape={snippet.joint_angles.shape}, dtype={snippet.joint_angles.dtype}" ) print( f" fwdkin_egoxyz: shape={snippet.fwdkin_egoxyz.shape}, dtype={snippet.fwdkin_egoxyz.dtype}" ) print( f" rawpred_egoxyz: shape={snippet.rawpred_egoxyz.shape}, dtype={snippet.rawpred_egoxyz.dtype}" )

Experimental data and metadata:
  legs: ['lf', 'lm', 'lh', 'rf', 'rm', 'rh']
  dofs_per_leg: [('thorax', 'coxa', 'pitch'), ('thorax', 'coxa', 'roll'), ('thorax', 'coxa', 'yaw'), ('coxa', 'trochanterfemur', 'pitch'), ('coxa', 'trochanterfemur', 'roll'), ('trochanterfemur', 'tibia', 'pitch'), ('tibia', 'tarsus1', 'pitch')]
  data_fps: 100.0
  experiment_trial: generated_from_ik_results
  framerange_in_raw_recording: [0, 100]
  joint_angles: shape=(100, 6, 7), dtype=float32
  fwdkin_egoxyz: shape=(100, 0, 3), dtype=float32
  rawpred_egoxyz: shape=(100, 0, 3), dtype=float32

Build the FlyBody model¶

The model uses leg-only articulation with position-controlled active leg DoFs, plus leg adhesion and the FlyBody-specific tendon actuators. The active DOF layout (42 position actuators) matches the standard locomotion fly, but here it is built on the FlyBody skeleton, axis convention, and neutral pose.

In [3]:

axis_order = FlyBodyAxisOrder.YAW_ROLL_PITCH
neutral_pose = KinematicPosePreset.FLYBODY_NEUTRAL
actuator_type = ActuatorType.POSITION

fly = FlyBody(name="flybody")
skeleton = FlyBodySkeleton(
    axis_order=axis_order,
    joint_preset=FlyBodyJointPreset.LEGS_ONLY,
)
fly.add_joints(skeleton, neutral_pose)

position_actuated_dofs = fly.skeleton.get_actuated_dofs_from_preset(
    FlyBodyActuatedDOFPreset.LEGS_ACTIVE_ONLY
)
fly.add_actuators(position_actuated_dofs, actuator_type, kp=100)
fly.add_tendons()
fly.add_tendon_actuators()
fly.add_leg_adhesion()
fly.colorize()

body_cam = fly.add_tracking_camera(
    name="body_cam",
    pos_offset=(0.0, -8.0, 0),
    rotation=Rotation3D("euler", (1.57, 0.0, 0.0)),
    fovy=35.0,
)

world = FlatGroundWorld()
world.add_fly(
    fly,
    [0, 0, 0.7],
    Rotation3D("quat", [1, 0, 0, 0]),
    bodysegs_with_ground_contact=FlyBodyContactBodiesPreset.LEGS_THORAX_ABDOMEN_HEAD,
    add_ground_contact_sensors=True,
)

sim = Simulation(world)
renderer = sim.set_renderer(
    [body_cam],
    camera_res=(240, 320),
    playback_speed=0.1,
    output_fps=25,
)

print("FlyBody model ready")
print("  position actuators:", len(position_actuated_dofs))
print("  tendon actuators:", len(fly.get_actuated_jointdofs_order(ActuatorType.TENDON)))

axis_order = FlyBodyAxisOrder.YAW_ROLL_PITCH neutral_pose = KinematicPosePreset.FLYBODY_NEUTRAL actuator_type = ActuatorType.POSITION fly = FlyBody(name="flybody") skeleton = FlyBodySkeleton( axis_order=axis_order, joint_preset=FlyBodyJointPreset.LEGS_ONLY, ) fly.add_joints(skeleton, neutral_pose) position_actuated_dofs = fly.skeleton.get_actuated_dofs_from_preset( FlyBodyActuatedDOFPreset.LEGS_ACTIVE_ONLY ) fly.add_actuators(position_actuated_dofs, actuator_type, kp=100) fly.add_tendons() fly.add_tendon_actuators() fly.add_leg_adhesion() fly.colorize() body_cam = fly.add_tracking_camera( name="body_cam", pos_offset=(0.0, -8.0, 0), rotation=Rotation3D("euler", (1.57, 0.0, 0.0)), fovy=35.0, ) world = FlatGroundWorld() world.add_fly( fly, [0, 0, 0.7], Rotation3D("quat", [1, 0, 0, 0]), bodysegs_with_ground_contact=FlyBodyContactBodiesPreset.LEGS_THORAX_ABDOMEN_HEAD, add_ground_contact_sensors=True, ) sim = Simulation(world) renderer = sim.set_renderer( [body_cam], camera_res=(240, 320), playback_speed=0.1, output_fps=25, ) print("FlyBody model ready") print(" position actuators:", len(position_actuated_dofs)) print(" tendon actuators:", len(fly.get_actuated_jointdofs_order(ActuatorType.TENDON)))

/home/sibwang/Projects/flygym/src/flygym/compose/fly/flybody.py:798: UserWarning: abdomen1 not found in skeleton; skipping abdomen tendon creation.
  warnings.warn(

FlyBody model ready
  position actuators: 42
  tendon actuators: 6

Replay the recording¶

The target joint trajectories are obtained through MotionSnippet.get_joint_angles, which handles smoothing, interpolation, and DOF reordering for the simulator. During the replay itself we keep the tendon actuators neutral.

In [4]:

sim_timestep = sim.timestep
position_targets = snippet.get_joint_angles(
    output_timestep=sim_timestep,
    output_dof_order=fly.get_actuated_jointdofs_order(actuator_type),
    sgfilter_window_sec=0.05,
)
print("Position targets shape:", position_targets.shape)

sim_timestep = sim.timestep position_targets = snippet.get_joint_angles( output_timestep=sim_timestep, output_dof_order=fly.get_actuated_jointdofs_order(actuator_type), sgfilter_window_sec=0.05, ) print("Position targets shape:", position_targets.shape)

Position targets shape: (10000, 42)

In [5]:

nsteps_sim = position_targets.shape[0]
n_dofs = len(fly.get_jointdofs_order())
n_position_actuators = len(fly.get_actuated_jointdofs_order(actuator_type))
n_tendon_actuators = len(fly.get_actuated_jointdofs_order(ActuatorType.TENDON))

simulated_joint_angles = np.full((nsteps_sim, n_dofs), np.nan, dtype=np.float32)
position_actuator_forces = np.full(
    (nsteps_sim, n_position_actuators), np.nan, dtype=np.float32
)

fly_name = fly.name
zero_tendon_inputs = np.zeros(n_tendon_actuators, dtype=np.float32)

sim.reset()
sim.set_tendon_actuator_inputs(fly_name, zero_tendon_inputs)
sim.warmup()
smooth_start_time = 0.1
smooth_start_timesteps = int(smooth_start_time / sim_timestep)

for step_idx in trange(nsteps_sim, desc="Replaying"):
    target_angles = position_targets[step_idx, :] * min(
        1.0, step_idx / smooth_start_timesteps
    )
    sim.set_actuator_inputs(fly_name, actuator_type, target_angles)
    sim.step_with_profile()
    simulated_joint_angles[step_idx, :] = sim.get_joint_angles(fly_name)
    position_actuator_forces[step_idx, :] = sim.get_actuator_forces(
        fly_name, actuator_type
    )
    sim.render_as_needed_with_profile()

print("Replay complete")
print("  replay steps:", nsteps_sim)
print("  actuator target shape:", position_targets.shape)

nsteps_sim = position_targets.shape[0] n_dofs = len(fly.get_jointdofs_order()) n_position_actuators = len(fly.get_actuated_jointdofs_order(actuator_type)) n_tendon_actuators = len(fly.get_actuated_jointdofs_order(ActuatorType.TENDON)) simulated_joint_angles = np.full((nsteps_sim, n_dofs), np.nan, dtype=np.float32) position_actuator_forces = np.full( (nsteps_sim, n_position_actuators), np.nan, dtype=np.float32 ) fly_name = fly.name zero_tendon_inputs = np.zeros(n_tendon_actuators, dtype=np.float32) sim.reset() sim.set_tendon_actuator_inputs(fly_name, zero_tendon_inputs) sim.warmup() smooth_start_time = 0.1 smooth_start_timesteps = int(smooth_start_time / sim_timestep) for step_idx in trange(nsteps_sim, desc="Replaying"): target_angles = position_targets[step_idx, :] * min( 1.0, step_idx / smooth_start_timesteps ) sim.set_actuator_inputs(fly_name, actuator_type, target_angles) sim.step_with_profile() simulated_joint_angles[step_idx, :] = sim.get_joint_angles(fly_name) position_actuator_forces[step_idx, :] = sim.get_actuator_forces( fly_name, actuator_type ) sim.render_as_needed_with_profile() print("Replay complete") print(" replay steps:", nsteps_sim) print(" actuator target shape:", position_targets.shape)

Replaying: 100%|██████████████████████| 10000/10000 [00:01<00:00, 8816.40it/s]

Replay complete
  replay steps: 10000
  actuator target shape: (10000, 42)

Show the replay¶

After the replay finishes, we can display the rendered video inline and inspect the runtime profile.

In [6]:

sim.print_performance_report()
sim.renderer.show_in_notebook()

sim.print_performance_report() sim.renderer.show_in_notebook()

PERFORMANCE PROFILE

Stage	Time/step (us)	Percent (%)	Throughput (iters/s)	Throughput x realtime
Physics simulation advancement	88	86	11387	1.14
Rendering*	15	14	68808	6.88
TOTAL	102	100	9770	0.98

* Note: 250 frames were rendered out of 10000 steps. Therefore, rendering time per image is 581 us.

flybody/body_cam

Tendon actuation demo¶

The FlyBody model incorporates tendons to more faithfully actuate certain body parts. Here we build a fly with all biological joints — including the abdomen — and use a tendon actuator to drive the abdomen while holding the position actuators fixed.

In [7]:

from flygym.compose import TetheredWorld

fly = FlyBody(name="flybody")
skeleton = FlyBodySkeleton(
    axis_order=axis_order,
    joint_preset=FlyBodyJointPreset.ALL_BIOLOGICAL,
)
fly.add_joints(skeleton, neutral_pose)

actuated_dofs = fly.skeleton.get_actuated_dofs_from_preset(FlyBodyActuatedDOFPreset.ALL)
fly.add_actuators(
    actuated_dofs,
    actuator_type=actuator_type,
    kp=1.0,  # just to silence the warning
)
fly.add_tendons()
fly.add_tendon_actuators()
fly.colorize()

tracking_cam = fly.add_tracking_camera(
    pos_offset=[-1, -5, 1],
    rotation=Rotation3D("xyaxes", (0.999, 0.044, -0.000, -0.012, 0.265, 0.964)),
)

world = TetheredWorld()
world.add_fly(
    fly,
    (0, 0, 0.0),
    Rotation3D("quat", (1, 0, 0, 0)),
)

sim = Simulation(world)
renderer = sim.set_renderer([tracking_cam])
fly_name = fly.name

from flygym.compose import TetheredWorld fly = FlyBody(name="flybody") skeleton = FlyBodySkeleton( axis_order=axis_order, joint_preset=FlyBodyJointPreset.ALL_BIOLOGICAL, ) fly.add_joints(skeleton, neutral_pose) actuated_dofs = fly.skeleton.get_actuated_dofs_from_preset(FlyBodyActuatedDOFPreset.ALL) fly.add_actuators( actuated_dofs, actuator_type=actuator_type, kp=1.0, # just to silence the warning ) fly.add_tendons() fly.add_tendon_actuators() fly.colorize() tracking_cam = fly.add_tracking_camera( pos_offset=[-1, -5, 1], rotation=Rotation3D("xyaxes", (0.999, 0.044, -0.000, -0.012, 0.265, 0.964)), ) world = TetheredWorld() world.add_fly( fly, (0, 0, 0.0), Rotation3D("quat", (1, 0, 0, 0)), ) sim = Simulation(world) renderer = sim.set_renderer([tracking_cam]) fly_name = fly.name

/home/sibwang/Projects/flygym/src/flygym/compose/fly/flybody.py:638: UserWarning: No actuator config found for joint c_thorax-l_haltere-pitch.
  warnings.warn(f"No actuator config found for joint {jointdof.name}.")
/home/sibwang/Projects/flygym/src/flygym/compose/fly/flybody.py:638: UserWarning: No actuator config found for joint c_thorax-r_haltere-pitch.
  warnings.warn(f"No actuator config found for joint {jointdof.name}.")

In [8]:

sim_timestep = sim.timestep
nsteps_tendon = int(2.0 / sim_timestep)
hold_position = np.zeros(len(actuated_dofs), dtype=np.float32)
n_tendon_actuators = len(fly.get_actuated_jointdofs_order(ActuatorType.TENDON))
tendon_drive = np.zeros((nsteps_tendon, n_tendon_actuators), dtype=np.float32)
if tendon_drive.shape[1] > 0:
    tendon_drive[:, 0] = 0.35 * np.sin(np.linspace(0, 8 * np.pi, nsteps_tendon))

sim.reset()
print("Tendon actuators:", n_tendon_actuators)

sim_timestep = sim.timestep nsteps_tendon = int(2.0 / sim_timestep) hold_position = np.zeros(len(actuated_dofs), dtype=np.float32) n_tendon_actuators = len(fly.get_actuated_jointdofs_order(ActuatorType.TENDON)) tendon_drive = np.zeros((nsteps_tendon, n_tendon_actuators), dtype=np.float32) if tendon_drive.shape[1] > 0: tendon_drive[:, 0] = 0.35 * np.sin(np.linspace(0, 8 * np.pi, nsteps_tendon)) sim.reset() print("Tendon actuators:", n_tendon_actuators)

Tendon actuators: 8

In [9]:

for step_idx in trange(nsteps_tendon, desc="Tendon demo"):
    sim.set_actuator_inputs(fly_name, actuator_type, hold_position)
    sim.set_tendon_actuator_inputs(fly_name, tendon_drive[step_idx])
    sim.step_with_profile()
    sim.render_as_needed_with_profile()

sim.renderer.show_in_notebook()

for step_idx in trange(nsteps_tendon, desc="Tendon demo"): sim.set_actuator_inputs(fly_name, actuator_type, hold_position) sim.set_tendon_actuator_inputs(fly_name, tendon_drive[step_idx]) sim.step_with_profile() sim.render_as_needed_with_profile() sim.renderer.show_in_notebook()

Tendon demo: 100%|███████████████████| 20000/20000 [00:01<00:00, 17234.12it/s]

flybody/trackcam