Camera

The Camera class defines how images should be rendered from a camera in the world model. Cameras are added dynamically to the model by calling .body.add() on any body of the MuJoCo model programmatically in Python. In this way, the XML file does not need to contain all the preset cameras. When instantiating a Camera object, the user has to specify:

• The attachment point: this can be a body or a site in the model.

• The name of the camera.

• The targeted fly names. Those flies will be the one(s) whose contact forces will be drawn.

In order to simplify Camera instantiation, we provide a set of predefined camera parameters in config.yaml. Those parameters are used if the camera name matches the name of one of those cameras. In case the camera name does not match any of the predefined cameras, the user can specify the camera parameters manually.

This new logic for cameras allows a more flexible definition of the cameras update rules. We propose three different camera update rules: - ZStabilizedCamera: The camera z position is fixed at a given height above the floor. - YawOnlyCamera: Only the yaw and position of the camera are updated. This smoothens camera movements during locomotion in tracked cameras. - GravityAlignedCamera: This camera updates its orientation based on the changes in the gravity vector. This camera is useful for tracking the fly orientation in the world frame.

The full API reference of the different type of camera classes is as follows:

class flygym.camera.Camera(attachment_point: dm_control.mjcf.element._AttachableElement, camera_name: str, targeted_fly_names: list[str] | str = [], window_size: tuple[int, int] = (640, 480), play_speed: float = 0.2, fps: int = 30, timestamp_text: bool = False, play_speed_text: bool = True, camera_parameters: dict[str, Any] | None = None, draw_contacts: bool = False, decompose_contacts: bool = True, decompose_colors: tuple[tuple[int, int, int], tuple[int, int, int], tuple[int, int, int]] = ((0, 0, 255), (0, 255, 0), (255, 0, 0)), force_arrow_scaling: float = float('nan'), tip_length: float = 10.0, contact_threshold: float = 0.1, perspective_arrow_length: bool = False, draw_gravity: bool = False, gravity_arrow_scaling: float = 1e-4, output_path: str | Path | None = None)

Bases: object

init_camera_orientation(physics: dm_control.mjcf.Physics)

Initialize the camera handling by storing the base camera position and rotation. This is useful for cameras that need to be updated during the simulation beyond the default behavior of the camera.

render(physics: dm_control.mjcf.Physics, floor_height: float, curr_time: float, last_obs: list[dict] | None = None) → ndarray | None

Call the render method to update the renderer. It should be called every iteration; the method will decide by itself whether action is required.

Returns:

np.ndarray

The rendered image is one is rendered.

reset()

save_video(path: str | Path, stabilization_time=0.02)

Save rendered video since the beginning or the last reset(), whichever is the latest. Only useful if render_mode is ‘saved’.

Parameters:

pathstr or Path

Path to which the video should be saved.

stabilization_timefloat, optional

Time (in seconds) to wait before starting to render the video. This might be wanted because it takes a few frames for the position controller to move the joints to the specified angles from the default, all-stretched position. By default 0.02s

class flygym.camera.GravityAlignedCamera(*args, **kwargs)

Bases: Camera

Camera that keeps the camera aligned with the original direction of the gravity while following the fly.

class flygym.camera.YawOnlyCamera(smoothing_window_length=20, *args, **kwargs)

Bases: ZStabilizedCamera

Camera that stabilizes the z-axis of the camera to the floor height and only changes the yaw of the camera to follow the fly hereby preventing unnecessary camera rotations. The yaw is smoothed over a window of the last smoothing_window_length yaws.

class flygym.camera.ZStabilizedCamera(*args, **kwargs)

Bases: Camera

Camera that stabilizes the z-axis of the camera to the floor height.

init_camera_orientation(physics: dm_control.mjcf.Physics)

Initialize the camera handling by storing the base camera position and rotation. This is useful for cameras that need to be updated during the simulation beyond the default behavior of the camera.