Bilateral control design (FACTR 2)
Partly implemented design proposal
This page is a design proposal that lays out the implementation steps. Stages 1–5 of §7 are implemented and pass the mock tests. Nothing has been verified on hardware. The settings that work are under "Force feedback" in Configuration. Configuration keys, commands and node names for stage 6 onward (sensor, next, gripper feedback, etc.) are proposals and do not work in the current roboloom.
Written 2026-10-27. Sources studied (code was read, not run):
| Source | Version |
|---|---|
| philiphan0109/factr2_next | 0df02c5 (2026-09-04) |
| FACTR teleop (first version, for comparison) | 7a07ab3 |
| FACTR: Liu et al., Force-Attending Curriculum Training for Contact-Rich Policy Learning, RSS 2025, arXiv:2502.17432 | |
| FACTR 2: Oh, Liu, Tao, Han et al., arXiv:2606.12406 v2 |
1. Chosen approach
There are two main kinds of bilateral control.
| Approach | Follower | Sent back to the leader | Needs |
|---|---|---|---|
| Force reflection (FACTR / FACTR 2) | Stays position-controlled; current-based position also works | An estimate of the joint torque the follower receives from outside | An external torque estimate (torque sensors, or estimation from current) |
Symmetric (position–position, robopy's bilateral_joint) |
Current-controlled; the same PD coupling on both sides | Coupling torque from the position difference | Gravity models of both arms, torque constants, current control on the follower |
roboloom adopts FACTR 2's force reflection, because:
- The follower stays position-controlled. The existing
FollowerSafety(range, speed, deadline and jump checks, and HOLD) keeps working as the safety mechanism. - Only external force is sent back. The operator does not feel the follower's friction or inertia.
- FACTR 2's NEXT estimates external torque from motor current on arms without torque sensors. It applies to Dynamixel robots such as Rakuda2.
- It does not prevent adding the symmetric approach later (§8).
2. FACTR principles
Leader control law
The leader motors run in current (torque) control and output this sum (paper Eq. 5):
τ = τ_feedback + τ_null + τ_grav + τ_friction + τ_limit
| Term | Equation (paper) | Purpose |
|---|---|---|
| Force feedback | τ_feedback = μ_f K_fp τ_ext − K_fd q̇ (Eq. 1) |
Return the follower's external torque to the leader. K_fp is the ratio of leader to follower maximum torque, μ_f a scaling-down factor, K_fd damping against oscillation in contact |
| Gripper | τ_h,t = α(−k_h I_g,t) + (1−α) τ_h,t−1, α = 0.1 (Eq. 2) |
Return the follower gripper current as force on the finger, with an exponential moving average |
| Null space | τ_null = (I − J⁺J)(−K_np (q − q_rest) − K_nd q̇) (Eq. 3) |
Pull the redundant DOF (7-axis arm) toward a rest posture without an end-effector force |
| Gravity | g(q) by RNEA (Eq. 4) |
Keep the arm from falling |
| Friction | Static: a dither that flips sign at low speed. Kinetic: μ_c sgn(q̇) + μ_v q̇ (appendix Eq. 7–8) |
Break stiction so the arm moves easily by hand |
| Joint limits | A potential that repels near the ends (appendix Eq. 9–10) | Push back at the ends of travel |
Hardware: the leader uses XC330-T288-T and XM430-W210-T (about USD 1,230), the follower is a Franka Panda. The loop runs at 500 Hz. The Franka's own external torque estimate τ_ext is used as is.
The paper has no stability or passivity analysis. The only stabilizers are the damping K_fd, the factor μ_f and the gripper smoothing.
Policy learning
The external torque τ_ext enters the ACT policy as one token. Early in training the images are blurred, and the blur decreases over time (Force-Attending Curriculum), so that the policy does not ignore force. Average success on unseen objects: images only 21.3%, images + force (no curriculum) 61.2%, FACTR 87.5%.
Teleoperation with force feedback raised task completion by 64.7% and cut completion time by 37.4% compared with no feedback. The paper does not compare policies trained on data collected with and without feedback.
3. FACTR 2 and NEXT principles
FACTR 2 extends FACTR to arms without torque sensors. Its core is NEXT (Neural External Torque estimation).
τ̂_ext = τ_m − τ̂_free(x_{t−H+1..t}), x_t = [q, q̇, q_cmd − q]
τ_mis the torque from motor current (K·I).τ̂_freepredicts the torque the motors output without contact. An LSTM (2 layers × 128) predicts it from the last H = 50 steps of joint position, velocity and tracking error. It learns gravity, friction, inertia and the position controller's response together.- Training data is about 10 minutes of contact-free motion. Training takes about 1 minute; inference runs at 100 Hz.
- Accuracy: L1 error against the Franka torque sensor in contact is 0.547 Nm, against 1.471 Nm for a disturbance observer.
- Limits: the model must be retrained for each robot. The absolute torque scale depends on the accuracy of
K.
Feedback to the leader (Eq. 8) is τ_feedback = K_fp τ̂_ext. The gravity, friction and null-space terms stay as in FACTR. Policy learning replaced FACTR's curriculum with FIRST, which resamples training data to favour the phases around contact. As policy input, τ̂_ext did better than raw current (0.567 vs 0.431 on NIST insertion).
4. The factr2_next implementation
factr2_next is the official implementation of NEXT. It also contains a teleoperation demo in which a Gello-style leader drives an AgileX Piper. All communication uses ROS 2 JointState.
| Part | Content |
|---|---|
| Leader | 7 × XC330-T288-T, 4 Mbps, current control (Operating Mode 0). Control at 500 Hz; commands to the follower sent at 100 Hz |
| Follower | Piper (CAN). MIT mode (joint PD); each received command is written at once. No interpolation or speed limit |
| External torque | Piper effort − LSTM prediction. Velocity is a 200 Hz finite difference. The estimate is smoothed by an exponential moving average (α = 0.2) |
| Contact | Hysteresis on the L1 norm of the estimate (contact at 2.0, released at 1.0) |
| Feedback | gate · gains · τ_ext − damping · q̇, clipped to ±max_torque |
| Gate | 1 while in contact and the estimate is fresh (within 0.2 s), otherwise 0. It follows with ramp_alpha = 0.35 per 500 Hz tick, never stepping |
From factr2_feedback.py:162-176:
target_gate = 1.0 if self.contact and self._fresh() else 0.0
self.gate = (1.0 - self.ramp_alpha) * self.gate + self.ramp_alpha * target_gate
tau = self.gate * self.gains * self.tau_ext
tau -= self.damping * leader_joint_vel
tau = np.clip(tau, -self.max_torque, self.max_torque)
The default gains are negative ([-0.06, -0.08, -0.08, -0.03, -0.03, -0.02]). The estimate is the extra torque the motors output, which opposes the external force. The negative gains appear to return a force in the direction of the external force to the leader (our reading).
Points to note:
- In the shipped configuration, force feedback, gravity compensation and dither are all disabled. Only the null-space term and gripper feedback are enabled.
- According to the README, the gate, ramp and smoothing were not used in the paper's experiments.
- The Piper has 6 axes and is not redundant, so the null-space term is a weak posture spring with a fixed diagonal matrix.
- The follower has no command deadline. If the leader stops, it keeps holding the last target.
5. What roboloom already has
| Needed | roboloom today |
|---|---|
| Leader current control and combined laws | JointDevice and control laws (free, position, spring, gravity_comp, impedance), replaceable per group |
| Gravity compensation | gravity_comp, with g(q) (mA) identified from measured holding currents. No torque constant needed |
| Joint limits | The range barrier of gravity_comp (barrier_mA_per_count) and the runaway check (max_velocity, out of range) |
| Stiction | Breakaway hold (hold_mA). The opposite of FACTR's dither: it keeps the arm where it is left |
| Damping | kd (mA per count/s) |
| Follower state | The follower node's state has positions, velocities, currents_mA and the target actually written, target (= q_cmd) |
| Recording | record_effort: true records observation.effort (current) |
| Follower control laws | control in follower.yaml applies control laws on the follower too (e.g. current-based position). Not verified on hardware |
| Safety | FollowerSafety, command deadlines, Bus Watchdog detection, leader goes limp on estop, release and quit |
Missing:
- External torque estimation (model residual, NEXT).
- A return path from the follower to the leader. The leader node currently receives only
session. - The feedback term on the leader (gate, ramp, deadline, limit).
- Gripper force feedback.
- Recording of external torque, and of training data for the estimator.
Rakuda2's arms have 6 axes plus a gripper per side and are not redundant, so the null-space term is out of scope for now. Dither has the opposite aim of the breakaway hold; it will be considered as a separate control law if needed.
6. Design
Units and signs
roboloom handles joint values in raw motor units (counts, mA). The gravity model is also identified in mA, absorbing the torque constant. Feedback follows the same approach.
- The estimator outputs a
residualper follower joint, in the follower's current unit (mA for Dynamixel). - The definition is
residual = measured current − predicted current in free motion. It is the extra the motor outputs against the external force. - The external force acts in the direction of
−residual. - The leader current is computed as follows.
signis the sign in the controller'sjointsmap.
i_feedback[leader j] = − k_j · sign_j · residual[follower j]
- A starting point for
k_j(leader mA / follower mA) is, like FACTR'sK_fp, the ratio of leader to follower maximum torque times a scaling-down factor. For Rakuda2 this includes the ratio of the XC330 and XM430/XM540 torque constants. - On robots with
torque_constant_Nm_per_Aset per joint, values are also shown in N·m. For a robot that reports external force in N·m, such as a Franka, the estimator declares unitNmandk_jis in mA/Nm. - Signs are checked on hardware one joint at a time (§7 stage 5).
Dataflow
leader ──sample──► controller ──command──► follower ──state──► estimator
▲ │ ▲ │
└──── feedback ──────┘ └────────────── estimate ──────────┘
| Node | Added inputs and outputs | Role |
|---|---|---|
estimator (new role estimators) |
Input <follower>/state (latest only); outputs estimate, status |
Compute residual and contact from the follower state. A separate process, so that heavy inference never disturbs the follower node's period |
| controller | Input <estimator>/estimate (latest only); output feedback |
Map follower residuals to leader joints through joints, apply gain, send with a deadline. Send zero while the follower is not ACTIVE or before alignment (any state other than following) |
| leader | Input <controller>/feedback (latest only) |
Add the feedback term to the control law. The deadline, gate, ramp and limits are applied on the leader every tick |
- Joint correspondence and signs are written only in the controller's
joints. No separate map for feedback. - Feedback messages carry an absolute deadline, like follower commands (the same shape as
make_command). - The path is a cycle (leader → controller → follower → estimator → controller → leader). Stage 1 confirmed in mock that dora-rs dataflows accept cyclic edges (the controller and follower already formed a cycle).
Writing the configuration
Users never write edges. As in roboloom today, the configuration YAML only lists nodes by role. runtime/flow.py derives the edges from each node's references.
# nodes of the configuration (config.yaml) with the option bilateral enabled
inputs:
leader: nodes/leader_bilateral.yaml # type: rakuda_mini
controllers:
controller: nodes/controller_bilateral.yaml
followers:
follower: nodes/follower.yaml # type: rakuda2_follower
estimators:
effort: nodes/effort.yaml # follower: follower
The pairing follows from references alone.
| Reference | Written in | Derived edges |
|---|---|---|
input: leader, follower: follower |
controller | leader/sample → controller, controller command → follower (as today) |
follower: follower |
estimator | follower/state → estimator |
feedback.estimator: effort |
controller | effort/estimate → controller, controller feedback → leader (the controller's input) |
The rule is closed per pair. Treating the two arms as separate followers, or putting pairs of other robots in the same configuration, only means listing one set of nodes per pair. For example, another follower with another leader would be written as below. The device names are examples and do not exist in roboloom today.
inputs:
leader: nodes/leader.yaml # type: gello_dynamixel (example)
controllers:
controller: nodes/controller.yaml # type: joint, feedback.estimator: effort
followers:
follower: nodes/follower.yaml # type: piper_follower (example)
estimators:
effort: nodes/effort.yaml # type: next, follower: follower
Capabilities devices declare
Whether a combination works is checked when the configuration is read. Nothing branches on the device type; each class declares values that are compared.
| Declared by | Value | Checked |
|---|---|---|
Follower (Follower) |
effort_signal: the force quantity in state (one of currents_mA, torques_Nm, external_torques_Nm) and its unit |
It matches the estimator's input. sensor works only with robots that provide external_torques_Nm |
Leader (JointInput) |
Joints that accept feedback (groups of a current-mode law with feedback) and their unit (mA, Nm) |
Every joint in feedback.gain accepts feedback |
| Controller | The unit of feedback.gain (leader unit / follower unit) |
The unit matches both sides' declarations; every feedback joint maps one-to-one in joints |
| Estimator | follower, joint_names, unit, the state fields it needs |
The follower exists and provides those fields |
The feedback term is a part shared by every current-mode law (subclasses of CurrentLaw), not only gravity_comp. Another leader using another law accepts feedback with the same configuration.
Messages
Messages carry joint names and units, so that differences between devices never get mixed up silently.
| kind | payload |
|---|---|
effort_estimate |
joints (follower), residual, unit, contact, valid, robot_t_ns (time of the source state) |
feedback |
joints (leader), effort, unit, deadline_ns, gate (the controller's requested gate) |
The leader drops messages whose joint names or unit do not match its own and shows why in status.
Proposed configuration
nodes/controller_bilateral.yaml (added to the existing joints and so on):
feedback:
estimator: effort # a name under estimators in the configuration
deadline_ms: 100
unit: mA/mA # leader unit / follower unit
gain: # only joints listed here feed back (follower joint names)
r_arm_sh_pitch1: 0.1
r_arm_el_yaw: 0.1
An arm group in nodes/leader_bilateral.yaml:
- group: right_arm
law: gravity_comp # any current-mode law
# ... existing gravity_model, tuning, range, etc.
feedback:
max_mA: 40 # limit on the feedback part; the total stays limited by max_current_mA
ramp_s: 0.3 # time to open or close the gate
kd: 0.0 # extra damping in contact
The implementation puts contact detection (contact_on / contact_off) in the estimator, not the leader, so that the recorded observation.contact and the decision used for the gate are the same value and the leader stays robot-independent. The controller asks for the gate only while "the estimate is valid, in contact, the follower is ACTIVE, and following"; the leader checks the deadline, unit and joints and moves the gate over ramp_s.
nodes/effort.yaml:
type: model_residual # none | sensor | model_residual | next (none and model_residual are implemented)
follower: follower
joints: [r_arm_sh_pitch1, r_arm_sh_roll, r_arm_sh_pitch2, r_arm_el_yaw, r_arm_wr_roll, r_arm_wr_yaw]
models: [../calibration/effort_gravity_right.yaml] # implemented as a list of per-arm models
smoothing_alpha: 0.2
contact_on: 30 # L1 norm of |residual|, with hysteresis (in the estimator, as implemented)
contact_off: 15
record: true # records observation.effort_ext / observation.contact
Estimator contract
Like followers, inputs and sensors, estimators live in a registry ESTIMATORS (roboloom/estimators/) that selects the class by type. Robot-independent estimators are implemented in roboloom/estimators/; those for one robot only live in roboloom/robots/<model>/ and register in the same registry.
class EffortEstimator(Protocol):
follower: str # the follower node it reads
joint_names: tuple[str, ...] # follower joints
unit: str # e.g. "mA" or "Nm"
needs: tuple[
str, ...
] # state fields used, e.g. ("positions", "velocities", "currents_mA", "target")
def reset(self) -> None: ...
def update(self, state: dict, t_ns: int) -> Estimate: ... # follower_state payload
@dataclass
class Estimate:
residual: list[float] # measured − predicted in free motion
contact: bool
valid: bool # False with too little history, a stale state, etc.
type |
Content | Use |
|---|---|---|
none |
Always valid=False |
Path tests, no feedback |
sensor |
The robot's own external torque estimate (e.g. Franka tau_ext) |
Robots with torque sensors |
model_residual |
currents_mA − g_f(q), with g_f identified on the follower by the existing regressor gravity model |
First baseline. Works without learning; friction and inertia remain in the residual |
next |
currents_mA − LSTM(q, q̇, target − q) |
The FACTR 2 method |
Robot-specific parts stay inside the estimator implementation and its configuration. The controller and leader see only Estimate.
Where models are stored
Measured values and trained weights are kept with the configuration of each physical unit in .roboloom/<robot>/ (the directory roboloom init <robot> creates), never in the package. As with today's gravity models, YAML is tracked in Git and large raw data and weights are excluded.
| Location | Content | Git |
|---|---|---|
calibration/effort_gravity_<side>.yaml |
The follower gravity model for model_residual (the existing regressor format) |
Tracked |
models/<estimator name>/<run_id>/model.yaml |
The NEXT manifest (table below) | Tracked |
models/<estimator name>/<run_id>/weights.safetensors |
Trained weights | Excluded (*.safetensors) |
logs/<run>/follower_trace.npz |
Free-motion training data (the stage-2 trace) | Excluded (existing *.npz, logs/) |
run_idis the training date and time (e.g.20261101-153000). The node YAML names the version explicitly; "latest" is never chosen automatically, so that which model ran stays reproducible.- Weights are stored as safetensors. Without pickle, loading never executes code.
torchbecomes an optional extra (next). - Datasets record the estimator's
typeandrun_idused while recording.
What the manifest (model.yaml) records:
| Item | Example |
|---|---|
| Format | format: 1, type: next |
| Robot | Robot type, node name, port (by-id, identifies the unit), joint names, motor IDs |
| Inputs | Features and order (positions, velocities, target - positions), units, rate_hz, history length, normalization mean and variance |
| Model | Architecture (e.g. LSTM 2×128), training settings |
| Data | logs/ paths used and the sha256 of each file |
| Evaluation | Error on validation data, checks with known weights |
| Provenance | sha256 of the weights, creation time, roboloom commit |
On loading, these are checked and startup is refused on any mismatch:
- The follower
type, joint names and motor IDs match. rate_hzequals the follower'scontrol_hz. NEXT's history length depends on the rate, so a changed rate needs retraining.- The sha256 of the weights matches.
portis the same unit. Using it on another unit requires an explicit setting that allows it.
The roboloom init template gains a description of models/ and *.safetensors in .gitignore.
Safety
- Feedback is disabled by default. Only joints listed in
feedback.gainfeed back. - On an expired deadline,
valid=False, followerHOLD,FAULTorESTOP, or before controller alignment, the gate closes to zero overramp_s, never at once. estop,releaseandquitmake the leader limp as before.- Feedback current is first limited by
feedback.max_mA. The total with gravity, hold and barrier is then limited by the existingmax_current_mA. - The EEPROM Current Limit is the only limit that remains when the host stops. On XC330 a fired Bus Watchdog does not stop the current. Raising the Current Limit for feedback is written with
commission, knowingly. - The existing runaway check (
max_velocity, out of range) stays active during feedback. - The follower stays protected by
FollowerSafety. With the follower in current-based position control (Operating Mode 5), contact force is also capped.
7. Implementation steps
Each stage is committed and passes mock tests before moving to hardware. Hardware checks are done by the user.
Status (2026-11-02): stages 1–5 are implemented and pass the mock tests (tests/test_bilateral.py, including the whole path from the estimate through the gate to the leader current on dora). The hardware checks of stages 3–5 have not been done. The stage 2 trace is <follower name>_trace.npz; stage 4 identification uses the existing identify-gravity on a follower calibration (calibration/follower_<side>.yaml).
| Stage | Content | Done when |
|---|---|---|
| 1. Path | Add the estimators role, the ESTIMATORS registry and the none estimator. Add capability declarations and their check when the configuration is read. Derive edges from references in flow.py; add the controller feedback output and the leader feedback input. Feedback values are always zero |
Tests in a mock dataflow cover the cyclic edges, the feedback deadline, latest-only delivery, and refusal of configurations whose units or joints do not match. Existing tests pass |
| 2. Follower recording | Add a per-tick trace to the follower node (positions, velocities, currents_mA, target, mode) in the same form as the leader's leader_trace.npz. Allow recording estimator output as observation.effort_ext and observation.contact |
The npz and dataset features are checked in mock |
| 3. Follower mode | Add a configuration example that runs the arms in current-based position (current_position, Mode 5) through control in follower.yaml. Write the mode and Current Limit with commission |
Hardware: tracking and holding match today's; current stops at the limit in contact |
4. model_residual |
Build a tool that measures holding currents on the follower and identifies g_f(q) with the existing regressor (identify-gravity for the follower). Implement the model_residual estimator |
Hardware: record residual size at rest and in free motion; check the residual under a known weight |
| 5. Leader feedback term | Add feedback as a part shared by current-mode laws: gate, ramp, deadline, contact hysteresis, max_mA, kd. Show feedback_mA and the gate in status and the trace |
Mock: tests ramp to zero on deadline expiry, HOLD and estop. Hardware: enable one joint with a small gain; check the sign and that it does not oscillate |
| 6. Gripper | Move the trigger to a new current-mode law (spring + damping + contact term, like factr2_next's compute_gripper_feedback_torque), driven by follower gripper current (Mode 5, Current Limit 54 mA) |
Hardware: the trigger gets heavier when gripping an object and returns when released |
| 7. NEXT | Recording procedure for free motion (the stage-2 trace), training (torch as an optional extra) and an inference node (running at the follower's rate), under the same contract as model_residual. Write weights and manifest as in "Where models are stored" and check them on loading |
Hardware: compare with model_residual under known weights and pushes. Document the training and inference commands |
| 8. Delay | Report feedback delay (from the follower state's time until the leader writes current) in status. For the rate, see §9 | Hardware: record the delay distribution; document it with the gain limits |
| 9. Docs | For each implemented stage, update Controllers, Robots, Reference and Quick start, and remove the corresponding part of this page | The strict docs build passes |
8. Supporting new robots and leaders
A robot added as a follower needs:
statewithpositions,velocities,currents_mA(or external force) andtarget. For robots withoutvelocities, the estimator takes differences.- One way to provide external force:
sensorif it has sensors, otherwisemodel_residualornext. - To run under position control. A torque-controlled robot also works if it accepts position targets within
FollowerSafety's range.
A device added as a leader needs:
- Current (torque) control on the joints that receive force. Inputs that only produce positions (e.g. VR) are used without feedback.
- Gravity compensation (
gravity_comp, or that device's gravity model). - A one-to-one leader–follower joint correspondence in the controller's
joints.
A leader and follower with different kinematics (different number or order of joints) cannot be mapped joint by joint. They need an estimate of the end-effector force F, turned into Jᵀ F on the leader. That is out of scope here; the same contract (an estimator that outputs end-effector force instead of residual) allows adding it later.
The symmetric approach (robopy's bilateral_joint) can also be added later as a control law shared by leader and follower. It needs follower current control, gravity models of both arms and torque constants.
9. Open issues
- Delay: Rakuda2's bus runs at 1 Mbps and reads the state in 4 ms (Fast Sync Read). With it and the skipped wait before writes, the leader and follower went from 60 Hz to 120 Hz, checked on the hardware (Configuration, Control rate). The feedback path goes through one follower period, estimation, the controller and one leader period, so the delay is expected to be a few periods (about 20–30 ms at 120 Hz). That is still slower than FACTR's 500 Hz, so start with the gate, ramp, damping and weak gains. Raising the bus to 4 Mbps would allow higher rates, but it means rewriting the EEPROM of every motor and is not done here.
- Estimation accuracy: friction and inertia remain in
model_residual. Fast motion produces residual without contact. The contact threshold suppresses it, but replacing it with NEXT is the plan. - Current limit: the leader's Current Limit must be raised by the feedback amount. More force remains when the host stops.
- Use in policies: roboloom's scope ends at recording external force. FACTR's curriculum and FIRST belong to training.
- Scope of this study: FACTR 2's user-study numbers appear only in figures and were not verified. factr2_next's README and code disagree in places (e.g. the README lists gravity compensation as a feature, but it is disabled in the shipped configuration).