Reproducing DreamWaQ: Blind Quadrupedal Locomotion
on the Unitree A1, from Paper to Deployable Policy
DreamWaQ [1] trains a blind quadrupedal locomotion policy whose actor receives, in place of any exteroception, a learned velocity estimate and a variational context latent inferred from proprioceptive history. We report an independent reproduction on the Unitree A1 — the paper's own platform — in Isaac Lab, against a deliberately strong baseline that shares the method's rewards, privileged critic, and training budget. We find that the method's headline claim reproduces, but only under its own evaluation protocol: on aggregate locomotion metrics DreamWaQ is statistically indistinguishable from a well-trained blind baseline, while under lateral push perturbations it survives 88.7% of 1.5 m/s impulses against the baseline's 60.2%, and it climbs stairs blind where the baseline hesitates. Reaching this result required correcting four defects that are invisible in the paper: two ways an estimator network can silently corrupt PPO's importance ratio (one of which is present in the main community implementation), and three mechanisms absent from the paper's reward table without which training collapses — a variance term misprinted as its square, infinite ray-cast returns at terrain boundaries, and the inherited legged_gym positive-reward clip. We further extend the reproduction toward hardware deployment: actuator dynamics identified from physical chirp-response data (8 ms transport delay; delivered torque 0.84–1.00× commanded, never above), left-right symmetry augmentation, and a raised command envelope. The final policy attains the highest terrain-curriculum level of any configuration tested under the hardest physics, survives 95% of 2.0 m/s lateral impulses, and exports to a deterministic, privilege-free TorchScript module with a specified robot-side interface.
1 Introduction
Learned locomotion controllers for quadrupeds increasingly rely on state estimators trained jointly with the policy. DreamWaQ [1] is a prominent example: a context-aided estimator network (CENet) consumes a short proprioceptive history and produces an explicit body-velocity estimate and a 16-dimensional β-VAE latent intended to encode unobservable context — terrain, contact states, disturbances. The actor consumes only a single proprioceptive frame plus these two estimates; a critic with privileged simulator state trains alongside. The paper demonstrates the approach on the Unitree A1 with strong push-recovery statistics and blind stair climbing.
No official implementation exists. Community reconstructions disagree with the paper and with each other, and the paper itself omits several load-bearing details. This report documents a from-scratch reproduction with three aims: (i) determine which of the paper's claims survive against an honest baseline; (ii) document the corrections required to make the published description trainable at all; (iii) carry the result past reproduction to a deployable artifact for a physical A1.
Contributions:
- A working Isaac Lab implementation of DreamWaQ on the A1, with unit tests pinning the failure modes we encountered.
- A characterization of a bug family we call importance-ratio corruption: any non-determinism or update-ordering error in the actor's estimator inputs silently degrades PPO. We show one member of this family exists in the most-used community implementation and explain why it is masked on that project's robot.
- Three corrections to the paper's reward specification, each derived from measurement rather than tuning, without which training fails catastrophically.
- A claim-by-claim evaluation against a baseline matched on rewards, critic privileges, and budget — showing aggregate parity but reproducing the paper's perturbation-robustness advantage.
- A deployment-oriented extension: actuator identification from physical chirp data, measurement-grounded domain randomization, symmetry augmentation, and a verified TorchScript export.
2 Background
DreamWaQ. The actor observes ot ∈ ℝ45 (base angular velocity, projected gravity, velocity command, joint positions and velocities, previous action). CENet encodes the last five frames of the same quantities and outputs (v̂t, zt) ∈ ℝ3×ℝ16; the actor input is [ot, v̂t, zt]. CENet trains by a hybrid loss: velocity regression against simulator ground truth, next-observation reconstruction through a β-VAE bottleneck, and a KL term. The critic observes ot plus true velocity, a 187-point height map, and the applied disturbance wrench. Policy, value, and estimator train synchronously with PPO — the paper positions this single-stage design against teacher–student distillation. Training uses a game-inspired terrain curriculum over six procedural terrain families and the domain randomization of its Table II.
Related implementations. We surveyed 31 public repositories; none are official. We used one [4] as a reading reference for Isaac Lab integration patterns. Its own report shows DreamWaQ slightly below its baseline on aggregate reward — a foreshadowing of our §5.1 — and documents a posterior-collapse workaround (fixed β = 0.5) that §4.1 shows to be doing more than its author intended.
3 Implementation
We implement CENet at the paper's dimensions (encoder 128×64, decoder 64×128, latent 16, ELU), the asymmetric actor-critic, and the auxiliary loss inside the PPO update of rsl-rl 3.0.1. The robot model corrects a +4.1 mm lateral center-of-mass bias present in the stock A1 asset (an asymmetric body makes an asymmetric gait optimal) and randomizes the center of mass per environment. All networks and losses carry unit tests asserting shapes, gradient routing (the policy gradient must not reach the estimator; the velocity head must not train the decoder), and the determinism properties motivated below.
3.1 Importance-ratio corruption
Our first four full training runs collapsed identically: the terrain curriculum fell to zero and the policy converged to standing still, while the estimator itself trained well. The root cause is general enough to state as a rule:
PPO recomputes the policy on stored observations to form the importance ratio πnew/πold. Every input the actor consumes must therefore be a deterministic function of the stored observation, computed by the same parameters that generated the rollout. Anything else injects noise into the ratio that the trust region then spends compensating for.
We violated this twice. First, the actor consumed a freshly sampled latent z; each recomputation drew a different sample. The fix is for the actor to consume the posterior mean, confining sampling to the VAE loss. Second, the estimator took its ≈20 gradient steps before the PPO update within each iteration, so PPO recomputed the ratio through an already-moved network; the fix is ordering PPO first, against the exact estimator that produced the rollout. Figure 1 shows the training signature of each defect and fix.
The sampled-latent defect also exists in the community implementation [4]. Run verbatim on the A1, that code collapses within 60 iterations. It nevertheless works on its own platform (Unitree Go2) by an accident of scheduling: its fixed β = 0.5 collapses the VAE posterior (σ→0, hence z≈μ, hence effectively deterministic) — but only after ≈150 iterations. On the Go2 the policy survives until then; on the A1, which falls more readily, the curriculum dies first. A method whose viability depends on which of two pathologies wins a race is not robust; the deterministic-mean form removes the race.
3.2 Completing the reward specification
The paper's Table I lists twelve reward terms with weights. Implemented literally, training fails three distinct ways; each failure isolates a mechanism the paper uses but does not print.
| Observed failure | Cause | Correction |
|---|---|---|
| Policy immobile by iteration 40; one term contributes −3.9 per step against a tracking reward of +0.14 | The power-distribution penalty is printed as var(τ·θ̇)². At realistic per-motor powers the variance is 10³–10⁴; its square, at the printed weight 10⁻⁵, dominates all other terms combined | variance, not variance squared |
| All network weights become NaN at iteration ≈250 — precisely when the curriculum first promotes robots to terrains with edges | Height-scanner rays that miss geometry return +∞; the body-height and foot-clearance terms average ray heights | masked mean over finite rays only |
| Episode length collapses from 275 to 20 steps; robots fall deliberately, then converge to standing forever | An untrained policy earns net-negative per-step reward under Table I on any terrain; termination is not value-bootstrapped, so ending the episode quickly is optimal | clip total per-step reward at ≥0 — legged_gym's only_positive_rewards, a mechanism of the framework the paper builds on [3], inherited silently |
A fourth, quieter defect concerns posture. The stock Isaac Lab locomotion task — which most reproductions inherit — has no body-height term and zero weight on orientation; its optimum is a crouched shuffle, and every policy we and others trained on it crouches. Table I's height term fixes this in principle, but its printed weight (−1.0, quadratic) prices a 9 cm crouch at ≈0.5% of total reward — negligible against the stability value of a low stance once push perturbations enter training. The paper does not state its height target at all, and its sequel [2] randomizes the desired height, suggesting the printed table under-specifies the authors' actual practice. We raise the weight fivefold; Figure 2 traces the measured walking height across these configurations, including one of our own missteps (a constant-force disturbance model with no physical analogue) that drove the trunk below stair-riser height and made stairs geometrically unclimbable.
4 Experimental setup
Simulation. Isaac Lab 0.47 on Isaac Sim 5.1; rsl-rl 3.0.1 PPO with the stock A1 hyperparameters; 2048 parallel environments at 50 Hz control (4096 for the final run of §6); the standard six-family procedural terrain curriculum. Unless stated, runs are 3000 iterations ≈ 147M environment steps — 1.5× the paper's deployed-weights budget of 1000 iterations × 4096 environments.
Baselines. The privileged baseline gives the actor true body velocity and the height map — an upper bound, not deployable. The blind baseline is the comparison that matters: an actor seeing exactly one proprioceptive frame, with the same rewards, the same privileged critic, and the same budget as DreamWaQ. The paper's own baseline ("trained without any adaptation mechanism") is weaker on both counts; conclusions drawn against weak baselines were a recurring hazard in this work — the stock task's "blind" policy turned out to be privileged twice over, and every comparison against it overstated the architecture's contribution.
Evaluation. (i) Mean terrain-curriculum level, the standard aggregate; (ii) a push test following the paper's Table III protocol: 256 robots walking at 0.5 m/s receive a single body-lateral velocity impulse; survival = no fall within 2 s; five magnitudes, 0.5–2.0 m/s; (iii) measured walking height; (iv) single-robot follow-camera video on each terrain family (supplementary videos). One protocol correction is worth recording for future users of this test: all environments reset synchronously at evaluation start, so any post-push observation window that overlaps the common episode time-out reads as mass death. Our first sweep's 2.0 m/s column was an artifact of exactly this; the corrected protocol counts falls only.
5 Results
5.1 Aggregate performance: parity
At matched budget, rewards, and critic privileges, DreamWaQ and the blind baseline finish in a statistical tie: terrain level 5.54 vs 5.63, velocity tracking error 0.35 vs 0.35, with seed-to-seed spread of ±0.3–0.5 terrain levels (two seeds per configuration). The community implementation's report shows the same parity. This does not contradict the paper — the paper never ran this comparison — but it should calibrate expectations: on aggregate simulation metrics against an equal-privilege baseline, the architecture adds nothing measurable. Notably, the blind baseline tracks velocity commands at 0.35 m/s error with no velocity input of any kind; in clean simulation, a single noiseless joint-state frame evidently suffices, which relocates the case for an explicit velocity estimator to hardware conditions (IMU drift, slip) that simulation evaluation cannot arbitrate.
5.2 Perturbation robustness: the paper's claim reproduces
Under perturbation the picture inverts. Clean-trained DreamWaQ survives 100/99.6/88.7% of 0.5/1.0/1.5 m/s impulses against the blind baseline's 96.5/91.8/60.2% — a +28.5-point separation at 1.5 m/s between two policies that were indistinguishable in Figure 3. This is the paper's central claim, reproduced under its own protocol; our margins are smaller than the paper's (which reports 20% vs 95% survival) because our baseline is much stronger, not because the effect is weaker. Two further observations: disturbance training is worth more than architecture — 2000 iterations of annealed velocity kicks lift even the blind baseline to 100% at 1.5 m/s — and the two contributions compose, with the disturbance-trained DreamWaQ configurations extending survival to 95–96% at 2.0 m/s.
5.3 Blind stair traversal
Aggregate terrain level conceals per-terrain differences: both architectures saturate the easier terrain families, and stairs — the minority family — barely move the mean. Qualitatively the difference is consistent across checkpoints: the clean-trained DreamWaQ policy mounts stairs and places legs across risers (Video S1-class behavior appears from mid-training onward), while the blind baseline walks to the stair base and hesitates. The final policy climbs out of the evaluation stair pit reliably (Videos S1–S3). We did not quantify stair-specific success rates; the sequel paper's protocol (1,000 robots, success = summit within 10 s [2]) is the natural instrument and remains future work.
5.4 Ablations and negative results
| Claim | Verdict | Evidence |
|---|---|---|
| Estimator inputs improve push robustness over a blind policy | Reproduced | +28.5 points at 1.5 m/s, no disturbance training on either side (§5.2) |
| Near-oracle aggregate performance | Reproduced | Fig. 3 — though the blind baseline reaches it too |
| Blind stair climbing | Reproduced | §5.3, videos |
| Advantage on aggregate metrics vs an equal-privilege baseline | Not observed | §5.1 dead heat; the paper's baseline is weaker |
| Adaptive bootstrapping (AdaBoot) improves training | Not observed; harmful here | its gate pboot = 1−tanh CV(R) is highest at initialization, when reward variance is low because all agents are uniformly incompetent — feeding the actor an untrained estimate during the window in which the curriculum is decided. Disabled in all reported runs. |
| The context latent is beneficial per se | Conditional | withholding z from the actor improved a mid-budget configuration by ≈1 terrain level; an actively-training latent is a moving input the policy must chase, and pays for itself only after the estimator stabilizes |
6 From reproduction to deployment
The reproduction target was never the table above but a policy for a physical A1. Three additions close the gap between the simulated and physical robot.
6.1 Actuator identification from chirp response
We possess chirp-excitation logs from the physical robot: 500 Hz recordings of commanded position, measured position, velocity, and torque, swept 0.1→10 Hz at two PD gains under three payloads. A delayed-PD model — τ(t) = s·(Kp(q*(t−d) − q(t)) − Kdq̇(t)) — fits all 36 joint-recordings at R² 0.96–0.99 (Figure 5), yielding a transport delay of 8 ms (6–8 ms across joints) and a delivered-torque factor of 0.84–1.00.
Training then uses an actuator model with the physical robot's DC saturation plus a 5–10 ms command delay and per-episode gain scaling U(0.80, 1.05) — the two Table II randomizations we had not previously implemented, now grounded in measurement. The harder physics cost no measurable aggregate performance (Figure 3, final policy).
6.2 Left-right symmetry augmentation
Even on the symmetrized body we measured gait asymmetry growing over training: a limp is reward-neutral, so nothing opposes its emergence. We enable mirror-symmetry data augmentation (rsl-rl's native mechanism [5]): each PPO minibatch is doubled with its reflection across the sagittal plane. The A1-specific mirror maps — joint permutation (verified against the loaded articulation's joint ordering, which is breadth-first and differs from both the URDF and the vendor SDK ordering), sign conventions for vectors vs pseudo-vectors, and the height-map column permutation (derived programmatically from the simulator's own ray-grid construction) — are unit-tested for involution and equivariance.
6.3 The final policy
The final training run combines the corrected reward function, disturbance training (velocity impulses to 1.0 m/s, annealed), the measured actuator model, symmetry augmentation, and a command envelope extended to 1.5 m/s forward, at 4096 environments for 3000 iterations (≈295M steps, 3× the paper's budget; 1h51m wall-clock on one consumer GPU).
| Gate | Criterion | Result |
|---|---|---|
| Terrain level vs best prior configuration (5.82) | ≥ | 5.93, under the hardest physics tested |
| Walking height | ≥ 0.27 m | 0.290 m |
| Push survival (0.5→2.0 m/s) | ≥ prior best | 100 / 100 / 100 / 99 / 95% |
| Velocity tracking (commands to 1.5 m/s) | ≤ 0.40 | 0.36 |
| Gait symmetry | visual | videos S1–S14; residual asymmetry is the reader's call |
The exported artifact is a traced TorchScript module of the deployed path — one proprioceptive frame plus the five-frame history in, joint-position offsets out; deterministic (posterior means); privilege-free by unit test; 0.23 ms per step on a desktop CPU against the 20 ms control budget — together with an interface contract specifying observation layout, joint ordering, default pose, action scale, and PD gains for the robot-side control loop.
7 Discussion
What the paper's evidence structure implies. Our results align with a careful reading of the paper itself: its quantitative separations come from a push test and a cross-simulator tracking evaluation, not from aggregate sim metrics against strong baselines. Reproducers who benchmark estimator architectures on terrain-curriculum means should expect null results regardless of whether the method works, and should evaluate the claim the paper actually makes, under its protocol.
Threats to validity. Two seeds per major comparison (spread reported); the final policy is a single seed. All evaluation is in the training simulator; the cross-simulator and hardware halves of the paper's evidence remain untested here — the exported policy's rope test is the natural next step. The stair advantage is qualitative. AdaBoot's failure may be specific to our reward scales; we report the mechanism of failure rather than a universal verdict.
Lessons for reproducers. (i) Compare training-curve shapes, not endpoints — our decisive debugging clue was a curve that fell differently, not further. (ii) A reward table is not a reward function; when a printed weight produces a per-step magnitude that dwarfs the task reward, the paper means something it did not write. (iii) Build the strongest honest baseline first; we lost days to conclusions drawn against an accidentally-privileged one. (iv) Any input an actor consumes must be deterministic and update-ordered with respect to PPO's recomputation; estimator-augmented policies offer at least two easy ways to violate this, and both present as inexplicable curriculum collapse. (v) An afternoon of chirp data beat both of our guessed randomization ranges — and falsified one of the paper's.
8 Conclusion
DreamWaQ reproduces on its own robot, in the specific sense the paper claims: a proprioception-only policy whose estimator inputs buy substantial perturbation robustness and blind stair capability at zero deployment cost, with no teacher-student pipeline. It does not outperform an equal-privilege blind baseline on aggregate metrics, and its latent must be handled with care during training. With measured actuator dynamics, symmetry augmentation, and the corrected reward function, the approach yields a policy we consider ready for hardware trials on the physical A1.
Supplementary videos
Final policy
S1 — Stair ascent, medium difficulty, 0.5 m/s. Blind ascent out of the stair pit.
S2 — Stair ascent, high difficulty (approaching the training ceiling of ~23 cm risers).
S3 — Stair descent, the higher-fall-risk direction for a blind policy.
S4 — Incline at maximum training difficulty.
S5 — Box field, random height discontinuities.
S6 — Rough terrain at 1.5 m/s, the extended command envelope at its limit.
S7 — Stairs commanded at 1.5 m/s — whether the policy moderates its own stride.
Perturbation response
S8 — 1.5 m/s lateral impulses every 3–5 s while walking (measured survival 100%).
S9 — 2.0 m/s impulses (measured survival 95%).
S10 — Impulses during stair ascent, the compound case.
Training progression (for calibration)
S11 — Before the reward corrections: the crouched shuffle that the stock reward set optimizes for.
S12 — Corrected rewards (blind baseline): upright carriage restored.
S13 — Disturbance training before the height-weight increase: robust but stair-avoidant.
S14 — Final policy, last checkpoint.
Appendix A Deviations from the published method
| Deviation | Rationale |
|---|---|
| Actor consumes the latent mean, never a sample | importance-ratio correctness (§3.1); the paper's own robots likely behaved this way via posterior collapse |
| Policy update ordered before the estimator update | §3.1 |
| AdaBoot disabled | measured harmful; mechanism in Table 2 |
| Power-distribution term: variance, not variance² | Table 1 |
| Total per-step reward clipped at ≥0 | Table 1; inherited framework mechanism |
| Height term weight −5.0 (printed: −1.0); target 0.32 m (unprinted) | §3.2 |
| Disturbances: annealed velocity impulses ≤1.0 m/s; no constant wrench | annealing is ours (full-strength impulses from iteration 0 pin the curriculum at zero); payload randomization already covers Table II's payload as a mass |
| Motor strength U(0.80, 1.05); actuation delay 5–10 ms | measured from the physical robot (§6.1); the printed symmetric range is falsified by measurement |
| Left-right symmetry augmentation | anti-limp; not in either paper |
| Forward command cap 1.5 m/s (paper: 1.0) | deployment utility |
Appendix B Artifact inventory
| Path (repository-relative) | Contents |
|---|---|
artifacts/deploy/s2_deploy_final.pt + _contract.md | deployable TorchScript and robot-side interface specification |
artifacts/raw/train_*.log | per-iteration metrics for all 30 training runs |
artifacts/raw/push_eval_*.json | push-test results with full protocol parameters |
artifacts/raw/pace_actuator_fit.txt | the 36 chirp joint-fits of §6.1 |
artifacts/renders/ | ≈80 videos: per-configuration gaits, training-progress series, final gallery |
tests/ | unit tests: network wiring, gradient routing, determinism, symmetry involution/equivariance, export parity |
References
- [1] I. M. A. Nahrendra, B. Yu, H. Myung. “DreamWaQ: Learning Robust Quadrupedal Locomotion With Implicit Terrain Imagination via Deep Reinforcement Learning.” ICRA 2023. arXiv:2301.10602.
- [2] I. M. A. Nahrendra et al. “DreamWaQ++” (context-aware exteroceptive extension). IEEE T-RO 42:819–836. arXiv:2409.19709.
- [3] N. Rudin, D. Hoeller, P. Reist, M. Hutter. “Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning.” CoRL 2021 (legged_gym).
- [4] romankalyna. “dreamwaq-isaaclab.” github.com/romankalyna/dreamwaq-isaaclab (community implementation; reading reference).
- [5] M. Mittal et al. “Symmetry considerations for learning task symmetric robot policies.” ICRA 2024 (the rsl-rl symmetry mechanism).
Report generated August 15, 2026, from the repository's raw logs and evaluation artifacts. Videos are relative links; open this file from within the repository (directly, or via python3 -m http.server at the repository root).