RL training environments for Microduck (mjlab)

RL training environments for Microduck — a ~800 g, ~25 cm tall bipedal robot — built on mjlab (MuJoCo Warp) with PPO. Policies are trained here at 50 Hz, exported to ONNX, and deployed on the real robot by the runtime in pollen-robotics/microduck.

The repo encodes the full sim2real recipe: BAM actuator physics, domain randomization, backlash simulation, and the reward-design lessons that made it work (see AGENTS.md for the distilled playbook).

Requires a CUDA GPU (training runs through MuJoCo Warp) and uv.

On ARM boxes (DGX Spark / GB10, Jetson): uv sync pulls ~2 GB of CUDA wheels on first run and uv's default 30 s HTTP timeout can abort mid-download. Export UV_HTTP_TIMEOUT=600 for the first sync.

git clone https://github.com/pollen-robotics/microduck_rl
cd microduck_rl

# train the walking policy (uses your GPU; ~1-2 h for a usable gait at 4096 envs)
uv run train Mjlab-Velocity-Flat-MicroDuck --env.scene.num-envs 4096

# watch a trained policy in the viewer
uv run play Mjlab-Velocity-Flat-MicroDuck --wandb-run-path <entity/project/run_id>

# export to ONNX for deployment
uv run scripts/export.py Mjlab-Velocity-Flat-MicroDuck --wandb-run-path <...>

# drive the exported policy in CPU MuJoCo with the keyboard
uv run scripts/infer_policy.py --walking output.onnx

Resume from a checkpoint:

uv run train Mjlab-Velocity-Flat-MicroDuck --env.scene.num-envs 4096 \
    --agent.run-name resume --agent.load-checkpoint model_29999.pt --agent.resume True

No GPU? Add --hf-jobs to any train command to run it on Hugging Face Jobs instead of locally (see scripts/hf/README.md).

uv run list-envs prints the live registry. Flat/Rough variants exist where noted.

At deployment the runtime hot-swaps these policies (walk / recover / trick) behind a shared 61-dimensional observation contract, so any of them can take over the robot at any moment. scripts/infer_policy.py rehearses exactly that:

uv run scripts/infer_policy.py --walking walk.onnx --standing stand.onnx \
    --sitstand sitstand.onnx --roulade roulade.onnx --new-cmd-obs

Keyboard-driven (velocity commands, G ground pick, Y sit/stand, R roulade, K/L kicks); --debug, --save-csv, --record support sim2real comparisons.

Every main task has a Backlash twin that trains on a model with ±1° of gear play (2° total) in series with each of the 14 servo joints: insert -Backlash before MicroDuck in the task id, e.g. Mjlab-Velocity-Flat-Backlash-MicroDuck.

The backlash is modeled properly for sim2real: each servo gets an unactuated passive_<joint>_backlash hinge, and because the real encoder sits on the output side of the play, both the firmware PD emulation (BacklashEncoderBamActuator) and the joint_pos/joint_vel observations read through the backlash (qpos[servo] + qpos[backlash]). Observation and action dims are unchanged, so ONNX export and the runtime need no changes. See src/mjlab_microduck/tasks/backlash.py.

All tasks use the BAM M6 actuator model for the Dynamixel XL330 (voltage control law, back-EMF, Coulomb/Stribeck/load-dependent friction), with per-env domain randomization on battery voltage, voltage sag under load, command delay, and friction magnitude (FrictionDRBamActuator in src/mjlab_microduck/actuator/).

At this scale — tiny servos driving a ~800 g biped — actuator fidelity is most of the sim2real gap, which is why the actuator is modeled down to its voltage control law instead of an ideal PD.

MJCF models live in src/mjlab_microduck/robot/microduck/ and are exported from Onshape with onshape-to-robot, one config_mjcf_*.json per model:

scene*.xml files wrap the robots with a floor + keyframes (STAND/SIT/FOLD) for quick viewing and for infer_policy.py.

src/mjlab_microduck/
├── robot/
│   ├── microduck/                    # MJCF exports, export configs, scenes, add_backlash.py
│   └── microduck_constants.py        # robot cfgs, HOME frame, BAM actuator cfg
├── actuator/friction_dr_bam.py       # BAM + friction DR + backlash encoder feedback
├── tasks/
│   ├── __init__.py                   # task registration (base + backlash variants)
│   ├── mdp.py                        # rewards, events, observations, custom classes
│   ├── backlash.py                   # make_backlash_variant() env-cfg wrapper
│   └── microduck_*_env_cfg.py        # one cfg module per task family
├── train_cli.py                      # `train` entry point (+ --hf-jobs)
└── hf_jobs.py                        # Hugging Face Jobs submission

Conventions worth knowing:

• The observation layout is shared across every policy (61-dim actor obs: 48 proprioception + commands [twist(3), head_pose(4), body_pose(6)]), which is what makes runtime policy hot-swapping possible. Envs that don't use a command slot zero-pad it rather than dropping it.
• Unactuated joints are all named passive_* (roller wheels, backlash hinges); actuators, joint observations and pose rewards select servo joints with ^(?!passive_).*.
• Domain-randomization toggles are ENABLE_* booleans at the top of each env cfg file.
• Joint layout (14 servos): 0–4 left leg (hip_yaw, hip_roll, hip_pitch, knee, ankle), 5–8 neck/head (neck_pitch, head_pitch, head_yaw, head_roll), 9–13 right leg.
• The exporter bakes the observation normalizer into the ONNX graph — always deploy ONNX produced by scripts/export.py, never a hand-converted checkpoint, or the policy sees unnormalized observations at runtime.

AGENTS.md documents the env-building workflow and the reward-design rules learned across the project (also aimed at AI coding agents working in this repo).

uv run --with pytest pytest tests/

CPU-only config-invariant and reward-function regression tests — they lock in joint-index mappings, reward sign conventions, and NaN guards.

• microduck — the Microduck project home, including the onboard runtime that runs the exported policies
• mjlab — the training framework (MuJoCo Warp + rsl_rl)
• BAM — better actuator models, by Rhoban

This project is licensed under the Apache 2.0 License. See the LICENSE file for details. 3D model files are licensed under Creative Commons BY-SA-NC.