Evaluation and replay
How to keep whether each policy run succeeded, and how to replay recorded actions through the policy path.
Recording an evaluation's verdicts
Runs saved with roboloom infer --task <text> (record in Python) go into the evaluation dataset. Whether each run succeeded can be kept in the episode records (meta/roboloom_episodes.jsonl in the dataset).
uv run roboloom infer --seconds 30 --episodes 10 --task "pick up the cup"
uv run roboloom label --with policy 3 failure --note "missed the grasp" # afterwards, for episode 3
uv run roboloom check-dataset --with policy # labels: 7 success, 3 failure, ...
- A verdict can also be given when saving:
save successorsave failurein the operator screen's input,roboloom session save successfrom another terminal,record(..., label=...)orsave(label=...)in Python ("From Python" below). - To set or change a verdict afterwards, use
roboloom label [robot] <episode> success|failure(CLI).roboloom check-datasetcounts the verdicts (CLI). - The record also holds what the policy reported for the run (inference time
infer_max_ms, the steps asynchronous inference dropped as latedropped, its error) and the recorder's metrics. The keys are listed in Episode records. - A run in which the policy failed is not saved, so it has no record.
- LeRobot does not read this file; training is unaffected.
From Python
record(..., label="success")andsave(label="failure", note="...")save the episode with a verdict (successorfailure) and a note.recordalso takes a function aslabel: it receives the stopped episode (result) and returns the verdict. ReturningNonesaves without one.- Any other string raises
ValueErrorbefore anything is saved. - Every save appends one line to the dataset's
meta/roboloom_episodes.jsonl, with or without a verdict. - To set or change a verdict afterwards,
roboloom.recording.episodes.label(...)works too (Python API: recording).
from roboloom.recording import episodes
for k in range(10):
s.record(
"pick up the cup",
seconds=30,
label=lambda ep: "success" if cup_lifted() else "failure", # your own check
)
records = episodes.read(s.cfg.output) # {episode index: record}
print(episodes.summary(s.cfg.output, s.episodes))
# labels: 7 success, 3 failure, 0 not labelled (70% of the labelled)
Replaying a recording Mock only
roboloom.policy.replay:load sends the recorded action rows of one episode to the followers, whatever is observed. It runs the same path as a learned policy (controller, speed limit, interpolation, safety checks), so it checks that path without a model, and the replay's recorded observation.state can be compared with the original episode.
uv run roboloom infer rakuda2 --seconds 20 --home --task replay \
--policy roboloom.policy.replay:load \
--policy-kwargs '{"path": ".roboloom/rakuda2/datasets/rakuda2", "episode": 0}'
pathis a recorded dataset or a.npyfile[T, n]of actions (to check the joint order and units of data from another tool).stride: ktakes every k-th row (rate_hzmust bedataset_fps / k);horizonis the chunk length.- The dataset's action names and rate are checked against the configuration at load.
- It cannot be combined with
inference: async(refused at load): a replay returns its rows in order whatever is observed, so replacing the queued actions would skip recorded rows. - Move the robot to the episode's first pose before starting (home), and keep
max_speedlow on the first try. After the last action the followers stay there.