Cheng, Shi, Agarwal, Pathak. Extreme Parkour with Legged Robots. ICRA 2024. Started on the official Isaac Gym code, then built on CAI23sbP/Isaaclab_Parkour (GPL-3.0), a community Isaac Lab port.
The paper trains a quadruped to jump gaps, climb boxes and clear hurdles from a single depth camera. My A1 teacher reached the paper's curriculum level quickly, but walked with its rear feet on the floor. Finding out why took two weeks and several wrong theories. The answer was a missing reward. The fixed teacher crosses 0.8 m gaps in simulation. Its depth student, trained with depth noise fitted to the real camera, runs a full parkour course in MuJoCo: hurdles, stairs, box tops and ridges in one go. On the real robot the student only ever ran while hanging from a hoist.
Extreme Parkour trains in two stages. First a teacher policy learns with reinforcement learning, using privileged information a real robot doesn't have. It gets a grid of 132 terrain heights around the body and the heading error to its next waypoint, and a curriculum promotes it to harder gaps, hurdles and steps as it succeeds.
Then a student is distilled from the teacher by supervised imitation. The student sees a 58×87 depth image from the front camera at 10 Hz instead of the height grid. A recurrent encoder turns each frame into 32 numbers meant to match the teacher's terrain code, plus 2 that predict the heading, since a real robot has no waypoints. The actor runs at 50 Hz.
The teacher takes the same path with the encoder swapped for the true height grid and the true heading. Most of this page's trouble lives in what the teacher learned from that grid, and in the heading, which the student couldn't predict.
None of this is from scratch. In July I started on the authors' own code, chengxuxin/extreme-parkour, which already has an A1 configuration but no deploy code and no released weights. For deployment I took ideas from the community repos change-every/Extreme-Parkour-Onboard and its fork by Tang29Zx, which runs on a real Go2, and mined their issue trackers for known traps.
From July 18 I also ran CAI23sbP/Isaaclab_Parkour, a port of the paper to Isaac Lab with Go2 teacher and student tasks, and it became the main line. My main branch was that repo plus a minimal A1 swap and compatibility patches for Isaac Lab 2.3. Later, to rule out my own changes, I rebuilt it again as a1-minimal: the upstream code plus an 85-line robot swap and two compatibility shims. Training, terrain and PPO are upstream. I added reward terms, evaluation tools, a MuJoCo cross-check harness and the robot side.
The robot side is mine, in plain C++ on the A1's Jetson Xavier NX: a depth chain from raw RealSense frames to the encoder's input, the 8.9-million-parameter depth encoder and the actor as hand-written forward passes, a takeover state machine and safety watchdogs. On the NX the encoder takes 49–52 ms under load, against a 100 ms budget.
The community's known failure here is gaps: the student jumps them in Isaac Gym and walks into them anywhere else. I built a MuJoCo harness and ran three community checkpoints through it. They climbed stairs and steps but crossed 0 of 15 gaps, with no leap attempted.
The stock A1 recipe ran 15,000 iterations to terrain level 6.1 and leaped 0.8 m gaps. Its student fell apart in MuJoCo.
My first A1 teacher in the port walked on its front legs, from three config bugs. Fixing them gave a limp and a stalled curriculum. Mirror symmetry and two motion-prior (AMP) runs didn't fix the rear.
The deploy candidate that came out of July, go2teach_mts_61000, was a converged Go2 teacher distilled into an A1 student. I chose that route because the A1's own teacher dragged its feet. Measured on the clean A1 teacher: front feet peaked at about 0.9 cm, rear feet at 0.4 cm, against a target of at least 5.
On July 31 the cross-simulator gap closed in one line. Isaac Lab's camera returns Euclidean distance from the lens. MuJoCo's depth buffer, like a real RealSense, returns planar depth along the optical axis. They differ by 1/cos of the ray angle: 1.00 on axis, 1.38 at the edge of this field of view. Every student had been shown a systematically wrong image, worst at the edges where terrain features are.
| Same student, flat replica, 8 s | Travel | Speed |
|---|---|---|
| Isaac Lab (ground truth) | 6.15 m | 0.768 m/s |
| MuJoCo, planar depth | 0.43 m | 0.054 m/s |
| MuJoCo, corrected to Euclidean | 6.06 m | 0.758 m/s |
Five hypotheses died before this one. What found it was dumping an Isaac trace and its terrain, replaying the same terrain in MuJoCo, and diffing tensor by tensor: the depth diverged at step 0. The same correction matters on the robot, where the D435 sends planar depth: on real frames it shifts the encoder input by 0.2878. The community onboard repos rightly skip it, because the Isaac Gym lineage they deploy trains on planar depth.
In early August I ran numbered reward arms on the A1 teacher, one change each, graded on per-foot swing height (apex) and ground-contact share (duty).
| Arm | Change | Result |
|---|---|---|
| 3 | Torque penalty ×10 | −59% squared torque at equal terrain and tracking. Front apex 0.59 → 1.81 cm. Rear unchanged at 0.42 cm. |
| 5 | Clearance reward, flat only | Weight 0.5 did nothing; 2.0 lifted all four feet to about 9–10 cm. Flat only: on obstacles the rear-left got worse. |
| 6 | Clearance on all terrain | Flat fixed. On hurdle and gap the robot stopped using a leg entirely. |
| 7 | Clearance plus a duty floor | First policy to walk on four legs on all three terrains: every foot 7–10 cm, duty 0.46–0.59. |
| 8 | Arm 7 plus AMP style prior | Smoother, but terrain level 4.2 against 6.3. Rejected. |
| 9, 10 | Less clearance; tilt penalty everywhere | Drag came back; mixed. Arm 7 kept. |
Arm 6 was the instructive failure. The clearance term only charged a foot for being below its target height, so a leg held permanently in the air cost nothing. Arm 7's duty floor, a penalty on any foot loaded less than 20% of the time, closed that. It also showed every earlier policy had crossed obstacles on three legs, unnoticed.
On arm 8 I broke my own pre-registered kill criterion, 70% of arm 7's terrain at iteration 3,000. It came in at 65.1%, I let it run on a recovering trend, and the trend flattened at once. The log: "a pre-registered criterion is BINDING."
Arm 7's gait survived distillation into a depth student. But in MuJoCo that student's rear-right foot lifted 0.15 cm, against 9.57 cm in Isaac.
On August 9 I started over in an isolated clone to settle the question properly: why does the A1 teacher on this stack drag its rear feet? Every theory was tested on the same per-foot gait evaluation, at matched iterations and difficulty, and every one died.
A blind-walker test two days earlier had suggested a symmetric trunk model cut front-rear asymmetry by 2–3×. On the parkour teacher it didn't. Run 1, with the measured payload and a symmetrised trunk, was killed at about iteration 700: "already dragging rear". Run 2 used the original Isaac Gym body, symmetrised: rear apex 1.9 and 1.4 mm against 24–34 mm at the front. Centre-of-mass position didn't correlate with rear lift (r = 0.09 and −0.12), and disabling self-collisions changed nothing.
A scripted trot, no policy, identical front and rear commands. On the A1 the front feet lifted 31–36 mm and the rears 3.5–3.8 mm. On the Go2, which learns a balanced gait in this port, 30–32 front and 1.1–1.4 rear. The same 10× deficit on both robots. A Go2 trained under my protocol also dragged at iteration 200; it learns its way out by 50,000.
Run 4 went past 6,000 iterations, the furthest any A1 had trained here. The pathology didn't fade; it specialised. The rear-left dragged at 1.4–2.4 mm while the rear-right floated, flailing up to about 110 mm. Terrain sat near 5.3 from iteration 1,200.
In the probe, raising motor stiffness from kp 40 to 60–80 made rear lift symmetric. That evening I doubted it myself: "i don't think the open loop tests really mean anything." The script used a fixed ±0.3 rad swing. At kp 40 with a bigger swing, the rears lifted 40, 88, 171 mm. And on the run-4 policy the rear-right calf's commanded amplitude was zero. The physics could lift the feet. The policy never asked.
The paper's schedule is 50,000 iterations; every dragging A1 run had been cut at 1,500–13,000. On the minimal reproduction I ran A1, Go2 and Go1 at iteration 200 under the authors' configuration. All three dragged: rear apex 3.5/7.1, 2.8/4.5 and 3.0/4.4 mm. Nothing about the A1 was special. And a converged Go2 teacher loaded into the A1 body lifted its rears 39/40 mm from the first step.
From scratch, stock gains, plus a foot-clearance reward (target 6 cm, on all terrain) and the orientation penalty applied off flat ground too. The fronts lifted first, by iteration 800 the rears followed, and at 1,400 the run hit terrain level 6.89, the highest of any run. From about 2,800 on it held a front-rear symmetric gait with rears at 70–86 mm.
The conclusion, from the run log: "EP's reward set pays nothing for rear lift, PPO at short budgets converges to drag/collapse, and either paying for clearance directly or importing a mature gait skips the hole entirely."
The first sweep said both teachers refused trenches, reaching 60% of waypoints. That was the clock: in 10 seconds a robot at about 0.9 m/s covers half the 18 m course. And completions had been counted as falls. Under the paper's 30-second protocol both teachers crossed everything the terrain generator can make. Isaac Sim, 32 robots per cell, difficulty pinned, success rate:
| Obstacle | Size swept | Go2 warm-start | Run 8, from scratch |
|---|---|---|---|
| Gap | 0.1–0.8 m | 1.00 at every size | 1.00 at every size |
| Hurdle | 0.125–0.25 m (mid) | 1.00 | 1.00 |
| Step | 0.10–0.45 m | 0.97–1.00 | 0.97–1.00 |
| Stones | difficulty 0–1 | 1.00 | 1.00 |
| Stairs, never trained | 0.04–0.17 m rise | 1.00 | 0.97–1.00 |
True falls stayed under 1 per minute everywhere, highest on the tallest step. The teacher I distilled, the Go2 warm-start after a short fine-tune with centre-of-mass randomisation, beat the July deploy teacher on gaps, 1.00 against 0.72, and steps, 1.00 against 0.81.
On August 6 I ran the July student on terrains outside its training ranges. The teacher dropped 3% from trained to unseen terrain. The student dropped 61%. An ablation split the encoder's two outputs: keep its 32-number latent, but feed the true heading in place of its own prediction.
| Waypoints reached, of 1 | Teacher | Student, predicted heading | Student, true heading |
|---|---|---|---|
| Trained terrain, mean | 0.981 | 0.752 | 0.908 |
| Unseen terrain, mean | 0.949 | 0.291 | 0.864 |
| Cross-slope (camber) | 0.910 | 0.037 | 0.870 |
The collapse was two numbers out of thirty-four. On open ground a depth image holds nothing from which to infer the heading to a waypoint, so the prediction can't be learned. Raising its loss weight up to 64 times made camber worse. On the robot, the heading comes from the remote's stick. The operator supplies something the robot doesn't have.
A second ablation asked what was left. With the teacher's own latent swapped in for the student's, the narrow-beam terrain recovered 72% of its gap to the teacher and closely spaced thin gaps 57%. A gap-course control moved by 0.007. In the log's words: "the big residuals are perception, not policy." An auxiliary head that reconstructs the height grid from the depth latent looked like the cure, and on August 12 it lost: the baseline matched or beat it in 9 of 12 cells and doubled it on the unseen full staircase.
The last students, in late August and early September, came from a new series of from-scratch A1 teachers on the minimal reproduction, with depth noise fitted to real frames. Forty frames from this robot's D435 gave an invalid band on the left edge, noise of about 5 mm times distance squared, speckle holes, and sky that reads as zero (near), not far. Student 2 (model 61,998) had no falls on any of the six terrains in its final Isaac battery. In single-robot runs at iteration 60,000 it reached every waypoint on gap and step and 71% on hurdle. Its action jitter failed the target, 2.8–4.7 against 0.6.
The real test of that noise model is a different simulator. In MuJoCo, with the same degradation applied to every frame, student 2 runs a whole course: over hurdles, up a staircase onto a box, across box tops of different heights, down stairs, and over a row of ridges. It gets there on depth alone; the inset shows what it sees, holes and speckle included.
The parkour student ran on the A1 on August 5 and 6, always with the robot on a hoist. More on the robot is on the hardware page.
Camera pitch: 17.8 ± 0.7° nose-down, from a ground-plane fit in two poses, against the 20° in training. Too small to matter; I left it. Depth field of view: 59.43°, not the assumed 58. A chirp test on the hanging robot gave about 15 ms transport delay against the 20 ms in training.
Foot contact is in the observation, so a wrong channel is a constant, confident lie. Squeezing each foot in turn gave the order FR, FL, RR, RL. The scale is 0.3321 N per raw unit; my first figure, from a spec-sheet mass, was 21% wrong. The zero drifts by up to 18 raw units between sessions, so calibration became mandatory before arming.
"trying to trot. definitely rhythmic." All four legs locked to 2.34 Hz, close to the 2.36 Hz cadence in simulation. The rear calves were commanded past their joint limit 80.6% and 32.1% of the time. Weight-bearing, as I'd predicted in writing, that fell to 0.1% and 0.0%: hung, it never feels the ground and keeps reaching for it. But the fixed hoist held the robot back. "it just got stuck in a position where it just had FR foot out." No gait reading from that rig counts.
The full boot-to-drive sequence worked on the robot. The policy has no turn command; its only steering input is the heading error, which the student fills with its own guess. Outdoors there are no waypoints, so a biased guess is a steering bias. I wired the right stick into that slot instead.
55 51, not the documented FE EF. A magic-number check reported a working handset as dead.On September 11 I staged student 2 for the robot, with safety fixes ported back from the DreamWaQ deploy. On the hoist, commanding it to stand still pinned four of its twelve joints at the action limit and froze it there. It did exactly the same in MuJoCo on flat ground with the robot free to move, so the policy was at fault, not the port.
The cause was a configuration gap. The A1 student's settings inherited from the Go2 student, not from the A1 teacher. It never drew a zero command in training, and it never saw the terrain flag value the deploy loop writes. And the teacher itself held position on stairs by pawing, front feet in the air 90–96% of the time. The last commits, on September 13 and 14, fixed the student's command and body randomisation and narrowed standing to flat ground. There's no record of a student trained with those fixes running on the robot.
The same week, MuJoCo showed one more gap: with heading errors between 0.15 and 0.30 rad, student 2 fell in 6 of 6 trials. The deploy loop gained an optional band-stop that skips over it.
A side line put learned gait priors from APT-RL on this same parkour scaffold. It's covered on the APT-RL page.
These are the reports written during the work, unedited, as they were at the time.
Three-arm test on the blind walker with contact diagrams and video: a symmetric trunk cut front-rear imbalance. Superseded two days later, when the same body change on the parkour teacher still dragged.