Rakuda2: from collection to inference
Hardware-verified
- Goal: check the data collected by teleoperation, drive Rakuda2 with a policy trained with LeRobot, and keep each run's verdict.
- Prerequisites: you can collect with Rakuda2: teleoperated collection. To train on a GPU, check with
labstatusthat one is free. - Time: about an hour on top of collecting and training.
- Result: the evaluation dataset
datasets/rakuda2-evalholds episodes with verdicts.
What has been checked
On the real Rakuda2, teleoperated collection and policy inference (roboloom infer) are checked. roboloom.policy.lerobot:load, which reads LeRobot checkpoints, has been checked only on the mock PiPER + Pika Mock only. check-dataset and label are Mock only too.
flowchart LR
A["collect<br/>record"] --> B["check<br/>check-dataset"]
B --> C["train<br/>lerobot-train"]
C --> D["run<br/>infer --task"]
D --> E["verdict<br/>label"]
1. Collect
Collect episodes of one task as in Rakuda2: teleoperated collection.
uv run roboloom record --task "pick up the cup" --seconds 30 --episodes 20 --with gravity
gravitydoes not change the dataset name. The output is.roboloom/rakuda2/datasets/rakuda2.- To train on the DIGIT too, use
--with gravity,digit. The output becomesrakuda2-digit.
2. Check the recording
uv run roboloom check-dataset
- Check that
problemsisnoneand that theaction.valid is 0share undernotesis small. - To leave out bad episodes, see Excluding episodes.
3. Train with LeRobot
Mock only
uv add git+https://github.com/tetsugo02/roboloom.git --tag v0.2.0 --extra policy --extra train
uv run lerobot-train \
--dataset.repo_id=local/rakuda2 \
--dataset.root=.roboloom/rakuda2/datasets/rakuda2 \
--policy.type=act --policy.device=cuda --policy.push_to_hub=false \
--trackio.enable=false \
--output_dir=outputs/train/act_rakuda2
--dataset.repo_idis the configuration'srepo_id;--dataset.rootisoutputseen from the robot directory.- The checkpoint is written to
outputs/train/act_rakuda2/checkpoints/last/pretrained_model. - To try only the path,
--policy.device=cpu --steps=200 --batch_size=8takes a few minutes (Training with LeRobot).
4. Run the policy
First check the inference path with the template's default policy roboloom.policy:hold, which keeps the arms where they are. Without --task nothing is saved.
uv run roboloom infer --seconds 10
Then run the trained checkpoint. With --task, each run is saved to the evaluation dataset.
uv run roboloom infer --seconds 30 --episodes 10 --task "pick up the cup" \
--policy roboloom.policy.lerobot:load \
--policy-kwargs "{\"path\": \"$PWD/outputs/train/act_rakuda2/checkpoints/last/pretrained_model\"}"
inferturns on the optionpolicyby itself. It uses no leader, so it cannot be combined withgravity.- Pass the checkpoint as an absolute path. A relative path is not resolved from the project root (the dataflow is started from a temporary directory).
- Keep
rate_hzinnodes/policy.yaml(15) equal to the trained dataset'sdataset_fps(15). - With the default
mode: stepthe policy returns one step at a time, soaction_horizonhas no effect (LeRobot policies). - By default the policy observes the joint state and
camera. If you trained with the DIGIT, add--with digitand setobserveto[camera, digit_left, digit_right](Choosing observations). - To start every run from the same pose, write
home:innodes/policy.yamland add--home(home).
5. Keep the verdicts
Enter save success or save failure in the operator screen when saving, or set them afterwards.
uv run roboloom label --with policy 3 failure --note "missed the grasp"
uv run roboloom check-dataset --with policy
check-dataset shows the counts, like labels: 7 success, 3 failure, .... The verdicts are kept in the dataset's meta/roboloom_episodes.jsonl and do not affect LeRobot training (Evaluation and replay).
Next
- Let a person take over when the policy is about to fail: Rakuda2: evaluating with interventions
- Write your own model: Running your own policy