The full paper, first pass
Architecture, environment and training pipeline in one commit. Reward experiments followed; a legged_gym-style reward floor "zeroed the reward signal" within 50 updates and was dropped.
MOVEone network, a narrow view
Songbo Li, Shixin Luo, Jun Wu, Qiuguo Zhu. MOVE: Multi-skill Omnidirectional Legged Locomotion with Limited View in 3D Environments. arXiv 2412.03353, 2024. Written from scratch.
MOVE trains one policy that walks in any direction, uses a single forward depth camera to jump and climb when it's moving toward what it sees, and falls back on feel when it isn't. Over nine days in September I rebuilt it for the A1 in Isaac Lab. The blind walking came out well and stair climbing got close to the paper's numbers. The jumping, which is what the paper is actually about, never appeared, and the last run was stopped before it finished.
Reached: Isaac sim
The idea
Cheap quadrupeds usually have one camera, and it faces forward. When the robot walks sideways or backward, it can't see where it's going. MOVE's answer is to train the policy so that, from its joint history and one forward depth image, it builds an internal picture of the surroundings it can't see.
The paper calls the structure a pseudo-siamese network. One branch, the standard encoder, sees only what the robot will have: ten frames of proprioception and the current depth frame. It outputs a velocity estimate and three latents. The VAE latent is trained to predict the next frame, the front latent to reconstruct a clean front depth image, and the contrastive latent z_c to match the other branch. That other branch, the surroundings encoder, exists only in training. It sees privileged depth on five faces of a cube around the robot (everything except the rear) and attends to it with queries made from the robot's state. Eq. 7 of the paper pulls the two contrastive latents together with a SimSiam-style loss: two negative cosines, each with a stop-gradient on its target. The actor reads the current proprioception plus the standard encoder's outputs, so nothing privileged reaches it.
Robot's inputsTen 45-number proprioceptive frames, one forward depth image.
Standard encoderVelocity, a VAE latent, a front-depth latent, a contrastive latent.
Surroundings encoderTraining only. Five cube faces of clean depth, matched to the contrastive latent by Eq. 7.
ActorCurrent frame plus encoder outputs, 12 joint targets.
The paper used a Lite3 quadruped in Isaac Gym and reports success rates for six skills in its Table I: high jumps up to 0.7 m, long jumps across gaps up to 0.9 m, stairs with risers up to 0.15 m, crawling under roofs as low as 0.2 m, blind crawling, and walking with the camera shifted.
What I built
There's no official code. The project went through two codebases, both mine.
The first, Sep 15–16, was a direct implementation of the whole paper: the encoders and losses, a radial terrain set, and my own training loop. An audit against Eqs. 1–7 found no contradiction in the observations, networks or losses. Training was another matter. Obstacle training produced robots that walked in circles. A fresh run on flat ground did produce a validated walker, below, but every terrain result after that hung on a chain of warm starts, and the walker crouched at 0.13 m against a standing height of 0.27 m. On Sep 16 I closed that line:
The next work restarts from the stock Isaac Lab A1 rough recipe as reproducible from-scratch recipes, adding MOVE pieces one recipe diff at a time.
The second, Sep 16–23, is a numbered recipe ledger. Recipe 0 is Isaac Lab's stock A1 rough-terrain velocity task, trained with rsl_rl, unchanged. Each later recipe adds one thing and must train from scratch with a single command. Along the way came history instead of true velocity, the measured A1 body from my parkour project, randomization, the depth camera, MOVE's latent split and surroundings encoder (the 6c series), MOVE's terrain (recipe 7), and PIE's reward, which MOVE cites (recipe 9).
Each recipe trains on 4096 robots and is graded by the same suite: four terrains, four directions, at 0.5 m/s. A trial succeeds if the robot gets 3.5 m out on stock tiles, or 5.0 m on MOVE tiles, before it falls.
The walker that worked
The one result I'd call validated is the flat-ground proprioceptive walker from the first codebase, checkpoint 1800. It uses no camera, no history and no estimator: a 45-input MLP actor. I checked it against eight commands (forward, back, left, right, two turns in place, two forward turns) on three held-out seeds, 768 trials on the GPU. There were zero falls. The worst squared XY tracking error was 0.017, and the worst yaw error 0.27 rad/s. It's not a pretty gait. The feet stay low and slip somewhat, and the report says so.
How it went
A walker, then a restart
The flat walker passed its gates. Terrain fine-tunes from it stalled on easy stairs, and a height map fed straight into the actor didn't help, so I restarted on the recipe ledger. By evening, recipe 1 (blind actor, height-scan critic) had matched the paper's stair numbers on risers up to 0.15 m. Recipe 2, which drops true velocity for raw history, collapsed in every forward cell. That pushed MOVE's velocity estimator ahead of everything else.
Depth, MOVE tiles, and a counting bug
Depth went in and made stock stairs worse. I found that my evaluator had marked robots 0 and 1 as the only ones that ever fell, so every earlier fall count was invalid. That morning was also the first time Eq. 7's loss sat at its floor of −1 within 35 updates. That was the contrastive collapse, below.
A dead camera feature
Blinding the camera left one arm's evaluation files byte-identical to the normal ones. The depth CNN ended in an ELU that early auxiliary gradients drove to saturation by update 50, so every image gave the same vector. Ending it in LayerNorm brought it back. The next arm, 6c-L, was the first where vision clearly paid: 60.9% forward stair ascent, and 38.8% with the camera blinded.
The camera had never refreshed
On MOVE tiles the camera was a net handicap. Then I found out why: every depth policy so far had seen only the first frame of each episode. More on this below.
Live camera, and the lie-down
Retrained with a live camera, 6c-L climbed stairs forward at 82.5% across all rows and 94.3% on the paper's range. Per-row data on MOVE tiles then showed that blocked robots lie down. Recipe 9 switched to PIE's reward to stop that. I stopped it at update 1044. At checkpoint 1000 it crossed 0 of 8 times on the easiest step row, and its ported reward turned out to have two lifecycle bugs. That afternoon the server GPU was handed to a different project.
The last run
I fixed the reward histories and started recipe 9 again on stock tiles, from scratch, for 3000 updates. At update 1289 its terrain level was 4.66, against 5.48 for the live-camera baseline at the same point. I stopped it at update 1319 to free the GPU for other work. No evaluation or clips exist for it.
Where I departed from the paper, and what broke
The owner's rule was to stay close to the paper's architecture, with simplified parts allowed only as diagnostics. By Sep 21 the latent split, the Eq. 6 losses, the surroundings encoder, the terrain set and the camera model were faithful. Three things weren't. The mainline standard encoder is an MLP, not the paper's tokens, self-attention and GRU. That version exists, but alongside the cube map it doesn't fit 4096 robots in 16 GB. The critic is an MLP, not the paper's attention critic. And the per-foot depth patches were never wired in. The failures below did the most damage.
The stale camera
Isaac Lab refreshes a sensor lazily, when its .data is read, and the frame counter only advances inside that refresh. My observation code checked the frame counter before reading .data, so the refresh never ran. Nothing threw: the images were noisy, correctly shaped, and never changed after an episode's first frame.
So every depth result before Sep 20 was about one glimpse at spawn. On stock tiles that glimpse shows the step the robot is about to climb, and it alone was worth 22 to 34 points on stair ascent. On MOVE tiles the robot spawns in an empty 12 m centre, so the glimpse was wrong for the whole episode, and the camera hurt.
Blocked robots lie down
On MOVE's step rings, from the 0.17 m row up, almost no robot fell and none got through. They timed out, and mean trunk height dropped to zero. The clips show the robot walking to the step and lying flat on its belly. Three choices of mine made lying down the stable answer, and none of them came from the paper. The stock velocity reward pays a blocked robot the same whether it lies still or tries. The crawl-friendly termination I'd added made a trunk on the floor legal. And the curriculum neither promotes nor demotes a robot that never moves.
The owner had an answer ready: a blind DreamWaQ climbs 20 cm stairs easily. Its reward includes a body-height target and a foot-clearance term, so a robot can't settle on its belly. Mine had neither, and that made it a walker problem, not a vision problem. Recipe 9 tried PIE's table instead, since that's the one MOVE cites. It didn't get the chance to show anything before it was stopped.
The contrastive latent collapses
The health check for Eq. 7 is the spread of the contrastive latent, logged as aux_zc_std. The paper uses it in its Fig. 5. A healthy 16-number latent sits near 0.25. In every Eq. 7 arm it fell to about 0.003, while the loss read a perfect −0.999. Across 28 runs the split was exact: every Eq. 7 arm collapsed and every InfoNCE arm, my diagnostic substitute, held at 0.23–0.25. Re-reading the PDF on Sep 21 turned up the likely culprit, and it was mine. I had added a batch normalization in front of the cosine. That can rescale a near-constant latent back to unit variance, so the loss is satisfied by matching amplified noise. The one arm without it held 0.138 at update 500, but it was killed at 595, and the rerun that would have settled it was stopped at 324 when the server GPU went elsewhere. It didn't hurt stair performance: a fidelity defect, not the bottleneck.
Results against the paper
All of this is simulation. Nothing from this project ran on the robot. The paper's numbers are for the Lite3 on its own tiles. The stair numbers below come from the live-camera 6c-L run on stock Isaac Lab tiles, restricted to rows 0–5, whose risers go up to 0.15 m, the paper's limit.
| Skill, success % | Paper | Mine, best |
|---|---|---|
| Stairs forward (ascent, rows 0–5) | 96.3 | 94.3 |
| Stairs lateral | 97.1 | 94.8 / 85.9 |
| Stairs backward | 99.2 | 68.2 |
| Stairs descent, all directions | — | 100 |
| Stairs on MOVE tiles, forward | — | 14.7 |
| Crawl forward / lateral / back | 91.8 / 84.9 / 89.3 | 80.3 / 72.2 / 64.1 |
| High jump (steps) | 99.7 | 0 above 0.10 m |
| Long jump (gaps) | 99.6 | 0 above 0.33 m |
With the live camera, blinding it cost 38 points on forward stair ascent and multiplied falls by six, so by the end the policy really was using vision. It just used it like a blind walker that happens to have a camera. The handover note from Sep 21 says it best:
We have replicated the part of the paper a good blind walker with a camera can do, and none of the part the paper is about.
One fair caveat: the paper's top rows, 0.7 m up and 0.9 m across, are more than twice the A1's standing height. Rows up to about 0.4 m are the fair comparison, and even there no policy of mine climbed a step above 0.10 m or crossed a gap wider than 0.33 m.
What's still wrong
- No jumping, and on MOVE tiles a robot that sees a wall still lies down in front of it. The corrected PIE reward never got a full run there.
- The contrastive latent collapses. Removing my batch centring is the untested fix.
- Backward stair ascent is 68.2% against 99.2%, on one seed with a large spread between seeds.
- Simulation only. Nothing has been checked in MuJoCo or on the A1.