Camera¶

The Camera class defines how images should be rendered from a camera in the world model. Note that the Camera class does not by itself add a camera to the MuJoCo model — the camera must have already existed and can be referenced by its name. The camera can be added in two ways:

By adding a camera to the MuJoCo model file. Refer to the section on camera in the MuJoCo XML reference for more information. As an example, the following XML code adds a tracking camera to the MuJoCo model:

<worldbody>
    <!-- ... other things ... -->
    <camera name="camera_top" class="nmf" mode="track" ipd="0.068" pos="0 0 8" euler="0 0 0"/>
</worldbody>

By calling .worldbody.add() on the root element of the MuJoCo model programmatically in Python. Practically, this can be done by extending an existing FlyGym Arena class and adding the camera in the __init__ method. For example, the following code adds a stationary bird’s eye camera to the MuJoCo model:

from flygym.arena import BaseArena

class MyArena(BaseArena):
    def __init__(self):
        super().__init__()
        self.birdeye_cam = self.root_element.worldbody.add(
            "camera",
            name="birdseye_cam",
            mode="fixed",
            pos=(20, 0, 20),
            euler=(0, 0, 0),
            fovy=60,
        )
        # ... other things ...

Once the camera is added to the MuJoCo model, it can be referenced by its name to initialize the Camera object with additional parameters assigned to it. These can include: rendering rate in frames-per-second (FPS), window size, play speed, etc. Furthermore, the Camera class has useful methods such as .save_video, .reset, etc. Logics such as rotating the camera in tilted terrain are also implemented in the Camera class. The .render method is the main point of entry for fetching images from the camera.

The full API reference of the Camera class is as follows:

class flygym.camera.Camera(fly: Fly, camera_id: str = 'Animat/camera_left', window_size: tuple[int, int] = (640, 480), play_speed: float = 0.2, fps: int = 30, timestamp_text: bool = False, play_speed_text: bool = True, draw_contacts: bool = False, decompose_contacts: bool = True, force_arrow_scaling: float = float('nan'), tip_length: float = 10.0, contact_threshold: float = 0.1, draw_gravity: bool = False, gravity_arrow_scaling: float = 1e-4, align_camera_with_gravity: bool = False, camera_follows_fly_orientation: bool = False, decompose_colors: tuple[tuple[int, int, int], tuple[int, int, int], tuple[int, int, int]] = ((255, 0, 0), (0, 255, 0), (0, 0, 255)), output_path: str | Path | None = None, perspective_arrow_length=False)¶

Bases: object

Camera associated with a fly.

Parameters:

flyFly: The fly to which the camera is associated.
camera_idstr: The camera that will be used for rendering, by default “Animat/camera_left”.
window_sizetuple[int, int]: Size of the rendered images in pixels, by default (640, 480).
play_speedfloat: Play speed of the rendered video, by default 0.2.
fps: int: FPS of the rendered video when played at play_speed, by default 30.
timestamp_textbool: If True, text indicating the current simulation time will be added to the rendered video.
play_speed_textbool: If True, text indicating the play speed will be added to the rendered video.
draw_contactsbool: If True, arrows will be drawn to indicate contact forces between the legs and the ground. By default False.
decompose_contactsbool: If True, the arrows visualizing contact forces will be decomposed into x-y-z components. By default True.
force_arrow_scalingfloat, optional: Scaling factor determining the length of arrows visualizing contact forces. By default 1.0 if perspective_arrow_length is True and 10.0 otherwise.
tip_lengthfloat: Size of the arrows indicating the contact forces in pixels. By default 10.
contact_thresholdfloat: The threshold for contact detection in mN (forces below this magnitude will be ignored). By default 0.1.
draw_gravitybool: If True, an arrow will be drawn indicating the direction of gravity. This is useful during climbing simulations. By default False.
gravity_arrow_scalingfloat: Scaling factor determining the size of the arrow indicating gravity. By default 0.0001.
align_camera_with_gravitybool: If True, the camera will be rotated such that gravity points down. This is useful during climbing simulations. By default False.
camera_follows_fly_orientationbool: If True, the camera will be rotated so that it aligns with the fly’s orientation. By default False.
decompose_colors: Colors for the x, y, and z components of the contact force arrows. By default ((255, 0, 0), (0, 255, 0), (0, 0, 255)).
output_pathstr or Path, optional: Path to which the rendered video should be saved. If None, the video will not be saved. By default None.
perspective_arrow_lengthbool: If true, the length of the arrows indicating the contact forces will be determined by the perspective.

Attributes:

flyFly: The fly to which the camera is associated.
window_sizetuple[int, int]: Size of the rendered images in pixels.
play_speedfloat: Play speed of the rendered video.
fps: int: FPS of the rendered video when played at play_speed.
timestamp_textbool: If True, text indicating the current simulation time will be added to the rendered video.
play_speed_textbool: If True, text indicating the play speed will be added to the rendered video.
dm_cameradm_control.mujoco.Camera: The dm_control camera instance associated with the camera. Only available after calling initialize_dm_camera(physics). Useful for mapping the rendered location to the physical location in the simulation.
draw_contactsbool: If True, arrows will be drawn to indicate contact forces between the legs and the ground.
decompose_contactsbool: If True, the arrows visualizing contact forces will be decomposed into x-y-z components.
force_arrow_scalingfloat: Scaling factor determining the length of arrows visualizing contact forces.
tip_lengthfloat: Size of the arrows indicating the contact forces in pixels.
contact_thresholdfloat: The threshold for contact detection in mN (forces below this magnitude will be ignored).
draw_gravitybool: If True, an arrow will be drawn indicating the direction of gravity. This is useful during climbing simulations.
gravity_arrow_scalingfloat: Scaling factor determining the size of the arrow indicating gravity.
align_camera_with_gravitybool: If True, the camera will be rotated such that gravity points down. This is useful during climbing simulations.
camera_follows_fly_orientationbool: If True, the camera will be rotated so that it aligns with the fly’s orientation.
decompose_colors: Colors for the x, y, and z components of the contact force arrows.
output_pathOptional[Union[str, Path]]: Path to which the rendered video should be saved. If None, the video will not be saved.

dm_camera: dm_control.mujoco.Camera¶

initialize_dm_camera(physics: dm_control.mjcf.Physics)¶

dm_control comes with its own camera class that contains a number of useful utilities, including in particular tools for mapping the rendered location (row-column coordinate on the rendered image) to the physical location in the simulation. Given the physics instance of the simulation, this method initializes a “shadow” dm_control camera instance.

Parameters:

physicsmjcf.Physics: Physics instance of the simulation.

render(physics: dm_control.mjcf.Physics, floor_height: float, curr_time: float) → ndarray | None¶

Call the render method to update the renderer. It should be called every iteration; the method will decide by itself whether action is required.

Returns:

np.ndarray: The rendered image is one is rendered.

reset()¶

save_video(path: str | Path, stabilization_time=0.02)¶

Save rendered video since the beginning or the last reset(), whichever is the latest. Only useful if render_mode is ‘saved’.

Parameters:

pathstr or Path: Path to which the video should be saved.
stabilization_timefloat, optional: Time (in seconds) to wait before starting to render the video. This might be wanted because it takes a few frames for the position controller to move the joints to the specified angles from the default, all-stretched position. By default 0.02s

set_gravity(gravity: ndarray, rot_mat: ndarray = None) → None¶

Set the gravity of the environment. Changing the gravity vector might be useful during climbing simulations. The change in the camera point of view has been extensively tested for the simple cameras (left right top bottom front back) but not for the composed ones.

Parameters:

gravitynp.ndarray

The gravity vector.

rot_matnp.ndarray, optional

The rotation matrix to align the camera with the gravity vector: by default None.