Sensors
A sensor is a node that captures observations such as camera or tactile images. The recorder aligns them to the dataset rows, and a policy gets those chosen with observe. Images are handled as HWC, 3-channel RGB. A sensor's fps is its capture rate; the configuration's dataset_fps is the recording rate.
| Sensor | type |
Output | Status |
|---|---|---|---|
| RealSense | realsense_rgb |
RGB camera image | Hardware-verified |
| DIGIT | digit |
Tactile RGB image | Hardware-verified |
| Audio | mel_mono, pcm_mono |
Mel spectrogram, PCM | In development |
| Pika wrist cameras | v4l2_rgb, realsense_rgb |
Fisheye and D405 RGB images | In development |
| Web camera | opencv_rgb |
RGB camera image | In development |
By robot
| Robot | RealSense | DIGIT | Audio | Pika wrist cameras | Web camera |
|---|---|---|---|---|---|
| Rakuda2 | Hardware-verified | Hardware-verified | In development | — | In development |
| PiPER + Pika Gripper | In development | — | In development | In development | In development |
| Pika Sense in the hand | — | — | — | In development | — |
Audio and the web camera can be added to any robot's configuration. The table that also covers the kinds of policies is the Support matrix.
Recorder entry
Every sensor needs an entry under max_age_ms in nodes/recorder.yaml, keyed by its node name. It is the oldest a sample may be, in ms, to be used for a dataset row; an older one is recorded as missing (valid 0; the value repeats the last valid sample, zeros only before the first one). Inputs and followers need one too. Without it, loading the configuration fails with max_age_ms.<name>: expected positive finite number (for example max_age_ms.digit_left: expected positive finite number).
max_age_ms:
leader: 200
follower: 200
camera: 200
digit_left: 200
digit_right: 200
The Rakuda2 template already has the entries for camera, digit_left and digit_right. When you add a sensor or rename its node, add or rename its entry as well. Entries for nodes that are not in the configuration are ignored, so the template keeps the entries of every option (such as digit_left without the digit option); a misspelled name is ignored in the same way, and loading then reports the correctly named node's entry as missing.
The codec of storage: video images (AV1 or H.264) is chosen with video_codec in the same nodes/recorder.yaml (Video codec).