Correction · supersedes the earlier version of this page

It stands there

An earlier version of this page claimed this policy had finally got the APT-RL architecture working, on the evidence that it was mixing gaits 44.5% of the time. It is mixing gaits while standing still. Measured forward speed: 0.097 m/s.

standing

Motion library, unclipped, gait_diversity −15

Iteration 600. Alternates trot and bound near the 50/50 the reward asks for, and does not go anywhere.

0.097m/s forward

0.63metres in 6 s

1.86terrain level

walking

Plain joint-position teacher, no motion prior

Iteration 1499. Same probe, same play configuration, same terrain — so the standing above is a property of the policy, not of how it was measured.

0.310m/s forward

5.02metres in 6 s

2.10terrain level

The gait output is a free action

gait_diversity penalises ψ̄, the mean gait selection over a 5 s buffer, for departing from 0.5. At Table S3's weight of −15 that is worth up to 0.075/step — larger than every positive term in the reward combined, which tops out at 0.042/step.

But the gait selection is just a bit the policy emits. It can be toggled at 2 Hz without the robot moving at all. So standing still and alternating that bit collects the full diversity reward, while simultaneously avoiding the collision, action-rate, torque and contact-velocity penalties that locomotion incurs. The only thing given up is velocity tracking, capped at 0.042/step — less than the diversity term alone.

Standing is not a failure to optimise. It is the optimum of the reward as written.

Every aggregate signal said it was working

Signal at iteration 600–670	Value	Reads as
Mean reward	−44 → −19	learning steadily
Mean episode length	946	never falling
Terrain level	1.86	clearing hard obstacles
Gait mixing	44.5% bound	selector working as designed
Decoder torque share	54%	prior actively used
Forward speed	0.097 m/s	standing still

Terrain level deserves particular blame. It is scored as distance travelled relative to the commanded speed — threshold = command_velocity × episode_length, level up past 0.8× that — so it is not an absolute measure of ground covered, and a slow policy under slow commands climbs it. It had been treated throughout this project as the one trustworthy cross-arm metric. It is not one.

Where this leaves the comparison

Arm	Terrain	Fwd m/s	Verdict
Plain joint-position teacher	2.103	0.310	walks — the reference
Motion library, clipped −1	1.885	—	converged, but gait selector economically dead
Motion library, unclipped −15	1.855	0.097	stands still and games the diversity term
Motion library, clipped −15	1.888	—	void — reward blackout
Random decoder control	0.000	—	collapsed, 5 of 6 attempts

Three settings of one reward weight have now produced three different degenerate policies: a reward blackout that zeroes the gradient, a weight so low the gait selector goes unused, and a weight high enough that standing still beats walking. The forward-speed column is the one that should have been present from the beginning.