Nahrendra, Yu, Myung. DreamWaQ: Learning Robust Quadrupedal Locomotion With Implicit Terrain Imagination via Deep Reinforcement Learning. ICRA 2023. No official code exists. Training stack written for this project; the network and PPO classes on the deployed line are copied from romankalyna/dreamwaq-isaaclab (MIT).
DreamWaQ walks a quadruped without cameras: a small estimator reads the last few joint and IMU readings and guesses the body's velocity and a summary of the ground. I trained it in Isaac Lab, wrote a C++ forward pass for the A1's Jetson, and took it outdoors two days after starting. Three weeks later the robot ran a 29.5-minute outdoor session with two policies switchable from the remote. Most of this page is about what broke on the way.
s2_deploy_final) climbing out of a stair pit in Isaac Sim at a 1.0 m/s command. It has no camera and no height map.A blind policy sees one frame of proprioception, 45 numbers: body angular velocity, gravity direction from the IMU, the command, joint angles and speeds, and its last action. It can't tell a stair riser from a slope, and it doesn't know how fast it is moving.
DreamWaQ adds a context estimator, CENet. From the last five frames (100 ms at 50 Hz) it outputs an estimate of body velocity, trained against the simulator's true velocity, and a 16-number latent trained as a variational autoencoder that must predict the next frame. That latent is the paper's "implicit terrain imagination": to predict what the legs feel next, it has to learn something about the ground. The actor gets the current frame plus both estimates. A critic sees privileged state (true velocity, a height scan, the disturbance force) during training only. It all trains in one stage of PPO, with no teacher and student.
I surveyed 31 DreamWaQ repositories on Aug 13. None is official and each deviates from the paper somewhere, so I checked every loss against the PDF.
symmetrise_trunk, which zeroes a +4.1 mm sideways centre-of-mass offset in the stock A1 model.Each of these was forced by a measured failure, not chosen in advance.
.*THIGH matches no A1 body, so every policy could kneel for free), a one-sided linear foot-clearance term, and later a stand-still term.s2_deploy_final (sim) on rough ground, shoved sideways at 2.0 m/s every few seconds.The stock A1 task reproduced my July result and its rear-leg drag. Symmetrising the trunk moved the limp to the other rear leg, and the asymmetry grew with training. My first full CENet run failed: terrain 0.00, reward 6.20 against the baseline's 19.55. The latent had collapsed.
Four arms, 1500 iterations each. RefMatch verbatim: terrain 0.00. With the latent mean: 4.97. My own stack: 3.65 with a live latent, 4.61 with it withheld. Then the baseline turned out to see true velocity and a height scan, neither of which a real A1 has. I rebuilt it as a blind control with the same privileged critic.
With the paper's rewards and the clip, 3000 iterations each: blind control 5.63, DreamWaQ 5.54. A dead heat. Under single sideways shoves with no disturbance training, DreamWaQ survived 88.7% at 1.5 m/s against 60.2%: the paper's central claim. A month later I found this tool measured each level on the survivors of the previous ones, which flatters the harder levels.
s2_deploy_finalRefMatch with the mean fix, paper rewards plus height and clearance changes, the measured actuator model, ramped shoves to 1.0 m/s, symmetry augmentation, commands to 1.5 m/s. 4096 environments × 3000 iterations, 1 h 51 min on one desktop GPU. Terrain 5.93, then the project best; walking height 0.290 m.
Hardware fixes had become a chain of fine-tunes (v2 → s2_deploy9_ft → 9c → 9d, the last being PRD13d, which went outdoors on Aug 17), and five thermal arms hung off its end, so no result could be attributed to the change that named it. From here every run starts from random weights. That is fragile, so every config gets a ~300-iteration smoke test: reward must leave zero within ~15 iterations and action noise must fall.
On Sep 4 the tall policy, commanded backwards from a standstill, pitched over backwards on the robot; MuJoCo never reproduced it. That and the curb hesitation went into five single-change arms. The backward deadband was the reward's: under the soft tracking kernel, standing still earns ~73% of the reward at a 0.3 m/s command. The tall 0.36 m stance was the thermal lever. Seed-to-seed spread was larger than the differences between configs.
s2_deploy_final was still the best stair climber on record. A sweep traced its climb to symmetry augmentation plus an active foot-clearance term, but that config (symfc) drew one collapse, one strong climber and several ordinary ones across seeds.
| run (sim, Kp 34 / Kd 0.8) | survive 3.0 m/s shove | climb 0.19 m steps | climb 0.23 m steps | mech. cost of transport |
|---|---|---|---|---|
s2_deploy_final | 38.7% | 80.6% | 45.7% | 0.979 |
symfc seed 1 | 77.0% | 77.8% | 41.3% | 0.570 |
symfc seed 3 | 55.5% | 63.1% | 7.2% | 0.590 |
symfc seed 4 | 75.4% | 74.8% | 25.0% | 0.603 |
symfc seed 5 | 83.2% | 50.4% | 0.0% | 0.562 |
symfc seed 6 | 95.7% | 67.4% | 4.4% | 0.585 |
Same config, five seeds (seed 2 collapsed): 0–41% on the tallest steps. Climb is the share of 1024 spawns that mount at least three steps; shoves are 256 robots, one boot per magnitude. Read a ladder as a trend, never a pair.
The A1's Jetson Xavier NX has no PyTorch, ONNX runtime or TensorRT. So the policy ships as a flat weight file and a hand-written C++ forward pass (292,255 parameters: CENet encoder, velocity and latent mean, actor), checked on the robot against a PyTorch test vector: max difference 7.1×10−7 on the first export.
Most hardware failures here were silent mismatches, not physics. Each export has a written contract: observation order and scaling, joint order (breadth-first, unlike the URDF and the SDK), default pose, action scale, gains. The first one had the rear thigh default wrong, and it didn't say the history is ordered by observation term, which is the mistake the C++ then made.
Before the first run I fixed three defects in the inherited loop. A NaN action passed the clip (both comparisons are false) and reached the motors; now it holds stance. The tilt abort didn't latch, so a robot on its side re-primed itself after 3 s; now it latches into damping. The stall watchdog was one-shot per session; now it re-arms. And the inherited -ffast-math build flag makes g++ 7.5 fold std::isfinite() to true, deleting the NaN guard; the binary now tests its own guard at startup and refuses to run.
Every step that can move a motor needs my explicit go-ahead, because even a dry run applies damping torque. In order: weights against PyTorch; observation tests (gravity, joint permutation, defaults, and since Aug 16 history order, using values that encode their own position); a passive remote check, which found both horizontal stick axes inverted against Isaac Lab; robot hung with damping only; hung with live gains; then the ground, on a rope.
Before installing an export I now run the real control binary against the real weight file with a deliberately bogus recovery path: it loads the actor, prints provenance and exits before touching the motors. On Aug 22 I had verified a new file with a freshly built test binary, and the service binary turned out to be an older build that couldn't read it.
Everything in this section happened on the real robot unless it says sim.
41.5 s under policy control. Standing, all four feet down 90–93% of the time; commanded stepping rose to 2.02 lift-offs per second. At 18 s something tipped it 37.7° with three feet off the ground and it recovered in under 2 s. Six silent defects fixed on the way, including the rear thigh default, an inherited ×0.25 gyro and ×0.05 joint-speed scaling my policy never trained with, inverted strafe and yaw, and a stance ramp that snapped the joints at 3.42 rad/s (0.14 after).
Seven runs, five engaged, at Kp 40 / Kd 1.0, beyond the 20–26.25 it trained on. The longest walked 18.5 s of 41.6. The latching tilt abort fired twice on real falls, as built. No limp: front duty factors 0.599 and 0.597, rear 0.469 and 0.450, so the rear legs swung more than the front.
Nose-diving and kicking its rear up. Replaying the logs through the policy showed CENet estimating +1.09 m/s vertical on flat ground. Five hypotheses died, from joint-speed noise to actuator rolloff, before the cause turned up: Isaac Lab keeps a history buffer per observation term, so CENet trained on five angular-velocity frames, then five gravity frames, and so on, while the robot sent five whole frames end to end. Same 225 numbers, different order. Nothing threw, and the robot walked on its 45 correct inputs. Read in training order, the logged estimate tracked the command (+0.728 against +0.67 m/s). The replay had matched the robot to 1×10−4 because it copied the same mistake.
"they all hit the knees on the ground"
The knee penalty had never matched an A1 body name. The feet were scuffing pads that can't be replaced, and raising the paper's foot-clearance term did nothing: it is two-sided and squared, so it pulls high feet down as hard as it lifts low ones. One-sided and linear, median swing height rose 21 mm. I also measured the motors giving 0.79 of commanded torque near standstill and 0.41 at 5–10 rad/s, far outside training.
PRD13d went on at Kp 28 / Kd 0.7. In run0042 the robot slammed a curb edge and jammed the rear-right calf against its stop; the policy then asked for about 2.4 rad of extension from a joint that couldn't move. 33–37 Nm for five control steps at near-zero speed, and the driver cut the motor. The rear-left calf took its load and quit 9.5 s later, with the robot still upright. My first explanation, end-stop hammering, rested on a statistic that was 95% post-failure samples; I retracted it. The stall watchdog sums all twelve joints, so one stalled joint drowned in the other eleven's 234–409 W. Motor temperature hadn't been logged; it has been since.
run0050, 866 s outdoors. The rear calves reached 66–67 °C, every hip and thigh 41–48 °C, and the rear-left calf's driver began cycling torque off and on as it cooled. The robot kept walking.
After a thermal residual policy (5–10% less calf torque at matched joint speed on hardware, not the 32% sim suggested), I deployed tall36_stand_spread_s1, from scratch, 0.36 m stance. run0078: 13 minutes, calves 44 → 59 °C, squared torque 16–30% below the earlier blind policy at matched joint speed, the best thermals on this robot. It struggled to mount a curb, and sim doesn't reproduce that.
The A1 has no status LED or buzzer, so the motors say which policy is selected: an outward lift of the front-left hip for slot 1, front-right for slot 2. Then run0086, 29.5 minutes outdoors with both. Arm 4 drove 493 s, worst pitch 0.47 rad; arm 5 seed 2 drove 1141 s and rode rough ground hard, 32 s past 20° of pitch and one 66° roll at full speed. Forward starts worked every time (0.1–0.8 s); arm 4 still stuck on 4 of 8 backward starts.
The robot stood about 1 cm lower than in sim: a steady −0.10 rad calf error, PD droop rather than heat. Feeding measured standing torques forward at half scale halved it (mean 0.041 rad), sign right first time. The same session showed the left calves hotter than the right (front 67 vs 65 °C, rear 58 vs 52 °C) in straight-line walking.
A multi-objective policy went into slot 1 on Sep 8; no session with it is logged. On Sep 24 I installed two height arms, srv_h32 and srv_h38, after the on-robot weight test; run0109 shows both loaded and the service driving from the handset.
The knees set the duty cycle: about 12 minutes of continuous walking before the rear calves need rest, about 20 for the fronts, and slowing down never reaches equilibrium. Heat tracks torque squared, not the mechanical power the reward penalised. Across the calves, |τ·ω| correlated with the measured rise at r = −0.954, τ² at +0.990; a bigger energy penalty would have pushed load onto the rear calves that were already failing.
I fitted a first-order thermal model per joint to this robot (walking, idle, a static stand, a 36-minute cooldown): 0.44 °C RMS on a sensor that reports whole degrees. The rear calves aren't weaker motors; they carry about 53% more τ², a property of the gait. And a stationary robot cools about 17% slower than the walking fit implied, so a calf needs about 6 minutes to cool from 66 to 50 °C, not 4.5.
Then I tried a thermal paper's barrier reward (arXiv 2603.01631). In MuJoCo it walked 1.75× the distance before 60 °C with a peak 10.1 °C lower, slightly faster. But does the policy act differently when hot? Barely: the best arm changed calf torque by −2.0% across a 29 °C swing. The benefit was a fixed, better gait, which a tall stance gives without any temperature input.
Alongside walking I built DreamRiser (arXiv 2306.12712, same authors), a recovery policy that takes over mid-fall and hands back to the walker. It is a sim result. On the robot it has only run listen-only: 45 s at zero gain, nothing sent to the motors, tilted by hand through its sequence until it correctly gave up. The logs hold no hardware stand-up.
The paper as written gave 0.0% on my first measurement, and two later headlines were tooling: the checkpoint-picking metric charged one arm for its own settle phase, and a "rights itself 98% from anywhere" result came from a harness with the wrong action anchor and no joint clip. As of Sep 26 the best, StandClip seed 2, stands from its back 85% of the time in Isaac and 96% in MuJoCo. Its seed-1 twin scores 0% and 4%. Getting off the back is a skill a run finds or doesn't.
The original target was DreamWaQ++ (arXiv 2409.19709), which fuses a point cloud with CENet so the robot sees stairs instead of finding them with its front feet. Plain DreamWaQ was its stage 2. I implemented every component on Aug 13, with a simulated ray-cast camera.
For a month no configuration made vision matter, partly because of my bugs: the simulated camera looked 90° left, the encoder read the history in the wrong order, and PPO's recompute re-ran perception with a zeroed memory. The first camera-dependent policy came on Sep 11: in MuJoCo it climbed 8 cm stairs 8 of 8 times with the camera and stopped at the first riser 8 of 8 without. A larger run climbed 14 cm Isaac stairs 90.2% sighted and 33.2% blinded.
A paper-routed version saw and walked badly, with 31% of episodes ending in a body strike. I shelved the line on Sep 11 ("it looks like it does see, but it walks super shit"). The real D435 has never been in the loop.
eyes_s1, the blind stack plus a point-cloud encoder, on an 8 cm flight with its simulated camera on.These are the reports written during the work, as they were at the time. One link was changed: in the thermals report, a link to a private page now points to the incident report here.
The reproduction write-up, from the importance-ratio bugs to the first export, written hours before the first hardware run. It describes the paper-dimension CENet, but the exported policy used the community network, and its push numbers came from the sequential tool.
The first temperature log: which joints heat, how fast, and where the driver starts cycling.
The curb strike that cut out two calf motors, and why the watchdog missed it.
A per-joint model fitted to a walking and an idle log; refitted that evening over four sessions.
Five single-change arms; its MuJoCo terrace row was later traced to feet clipping block edges.
That week's arms on shoves, curbs, thermals and cost of transport; its shove numbers are from the sequential protocol, inflated by up to 28 points.