Skip to content

Dataset features

Detail Description
— The columns (features) roboloom writes to a LeRobotDataset 0.7.0. Which columns exist depends on the configuration's followers, inputs, sensors and recorded estimators, and on the enabled options. roboloom.recording.features(cfg) lists them for a configuration (see Recording). How rows are assigned and how missing data is handled is in Data format.
— <follower>, <input>, <sensor>, <estimator> and <source> are node names in the configuration (follower, leader, camera, digit_left and so on in the Rakuda2 template).
— <source> is every recorded node (inputs, followers, sensors, estimators with record: true).
— A column shared by several nodes (observation.state and others) joins their values in configuration order. Its names tell which entry belongs to which node (follower.r_arm_sh_pitch1).
— Shapes are per row.

Observation

Column dtype shape Present when Meaning
observation.state float32 (joints of every follower,) there are followers Each follower's joint positions, in its unit. names are <follower>.<joint>
observation.effort float32 (joints of the followers with record_effort,) a follower has record_effort: true Joint current or torque, in the model's unit (mA for Rakuda2 and Pika, N·m for PiPER). One column cannot mix models with different units
observation.<input> float32 (joints,) there is a joint input (a leader and so on) The input device's joint positions; names are the joint names. Option policy removes the leader, so the column is absent; policy_teleop brings the leader back, so the column is present
observation.<input> In development float32 (10,) there is a pose input (Pika Sense) x, y, z (m), qx, qy, qz, qw, gripper (rad; 0 for devices without one), tracking (1/0), engaged (1/0)
observation.images.<sensor> video or image (height, width, 3) a camera or tactile image sensor (RealSense, DIGIT, ...) RGB image. The dtype is the sensor's storage (default video; image is Mock only). names are height, width, channels
observation.audio.<sensor> In development int16 (sample_rate / dataset_fps,) type: pcm_mono One row of PCM
observation.audio.<sensor> In development float32 (n_mels, 1 + sample_rate·window_seconds // hop_length) type: mel_mono Log mel power of the last window_seconds (low frequencies first)
observation.effort_ext Mock only float32 (estimated joints,) an estimator with record: true External effort (the estimator's residual, negated), in the estimator's unit. names are <follower>.<joint>
observation.effort_ext.valid Mock only float32 (number of estimators,) an estimator with record: true 1 when that estimator's estimate is valid
observation.contact Mock only float32 (number of estimators,) an estimator with record: true 1 when it detects contact
task string (LeRobot turns it into task_index) — always The episode's task
Detail Description
--- ---
— A row without a usable sample repeats that node's last valid value (zeros before its first sample), and observation.meta.<source>.valid is 0 (Missing data).
— A policy receives its observation with the same names and shapes (without observation.meta; Observations).

Action

Detail Description
— A configuration without followers (a hand-held device) has no columns starting with action.
Column dtype shape Present when Meaning
action float32 (joints of every follower,) there are followers The first command each follower sent in the row's interval [t_k, t_{k+1}), in the order and units of observation.state. A follower that sent none fills its part with the target it held (else its position). Joints listed in a policy's hold are there too; explicitly claimed hooks can change their held target
action.valid float32 (1,) there are followers 1 when every follower sent a command in the row
action.clamped float32 (1,) there are followers 1 when a limit cut down any follower's command
action.intervention float32 (1,) the policy has teleop: (option policy_teleop) 1 when a person drove the row with the leader (teleoperation and interventions)
action.filtered Mock only float32 (1,) a policy YAML hooks: entry has nonempty writes (factory or from: policy) 1 if any valid command in any follower’s row interval was changed by a hook, before speed/safety limits; OR over all commands, including policy teleop. No column without writing claims

Metadata

Detail Description
— Each node's times and validity. Times are in time.monotonic_ns() time.
Column dtype shape Present when Meaning
observation.meta.<source>.valid float32 (1,) per recorded node 1 when a sample captured within max_age_ms of the row time t_k is available (one sample may serve several rows); at 0 the row repeats the last valid value
observation.meta.<source>.capture_age_ms float32 (1,) per recorded node Time since the sample was captured, seen from the row time t_k (ms); 0 without a sample
observation.meta.<source>.ready_age_ms float32 (1,) per recorded node Time since the sample became ready, seen from t_k (ms); 0 without a sample
observation.meta.<source>.seq int64 (1,) per recorded node The sample's sequence number; −1 without a sample
observation.meta.<source>.capture_ns int64 (1,) per recorded node When the sample was captured; 0 without a sample
observation.meta.<source>.ready_ns int64 (1,) per recorded node When the sample became ready; 0 without a sample
observation.meta.<source>.receive_ns int64 (1,) per recorded node When the recorder received it; 0 without a sample
action.meta.<follower>.valid float32 (1,) per follower 1 when that follower sent a command in the row's interval
action.meta.<follower>.clamped float32 (1,) per follower 1 when a limit cut that command down
action.meta.<follower>.seq int64 (1,) per follower The command's sequence number; −1 without one
action.meta.<follower>.sent_ns int64 (1,) per follower When the command was sent; 0 without one
Detail Description
--- ---
— LeRobot adds its own columns too.
Column dtype shape Meaning
timestamp float32 (1,) Time since the episode's start (s)
frame_index int64 (1,) Row number within the episode
episode_index int64 (1,) Episode number
index int64 (1,) Row number in the whole dataset
task_index int64 (1,) The task's number

In training

Detail Description
— LeRobot gives *.valid columns no special meaning: rows whose valid is 0 are trained on like any other. To leave them out, see Checking the data and training and valid.
— LeRobot puts every column starting with observation. into the policy configuration as an observation feature and every column starting with action as an action feature. The models of policies such as ACT, however, read observation.state, observation.environment_state and the images: observation.meta.*, observation.<input> and observation.effort are not model inputs, even when the dataset has them.

Rakuda2 example

Detail Description
— The columns that change with each option in the Rakuda2 template (roboloom init rakuda2).
Option Columns added Columns removed
none (teleop collection) observation.state, observation.effort, observation.leader, observation.images.camera, action, action.valid, action.clamped, action.meta.follower.*, observation.meta.{leader,follower,camera}.* —
digit observation.images.digit_left, observation.images.digit_right, and their observation.meta.* —
policy — observation.leader, observation.meta.leader.*
policy_teleop observation.leader, observation.meta.leader.*, action.intervention —
pika In development observation.right_sense, observation.left_sense, and their observation.meta.* observation.leader, observation.meta.leader.*
--sensors camera,audio In development observation.audio.audio, observation.meta.audio.* —
bilateral Mock only observation.effort_ext, observation.effort_ext.valid, observation.contact, observation.meta.effort.* —