gaitkeeper 0.5.2 · teleop-walking-benchmark

G1 Port Audit

Why each Unitree G1 walking policy in rhoyn's teleop-walking-benchmark scores what it does. Each port was read from its own adapter code, then the benchmark's 90 s waypoint tour was run one change at a time: the policy as its authors configured it, the port's values, the harness taking the arms, the random arm walk, the punches.

rhoyn/teleop-walking-benchmark@106faa0runs 0–9, the benchmark’s own drawsMuJoCo, the benchmark's g1_29dof.xmlevidence L12026-10-10

What stops each policy

1 Port values: the port's values break it

2 Arm handoff: trained to drive its arms; the harness takes them

9 Arm walk: stands, but the random arm motion knocks it down

8 Punches: survives the arms, falls to the punches

5 Policy: falls under its authors' own config

5 Survives: survives every stage within 5 s of its own setup

10 not benched (reasons below)

Agreement with the benchmark

0.98 Pearson 0.97 Spearman 212/300 runs within 1 s

30 policies. Survival of the port under the full benchmark here, against the benchmark’s own MuJoCo runs with the same run ids: seed n here draws the benchmark’s run n (its waypoints, arm targets and punches, from its own random streams). The port contracts are read from the adapters, none written by hand. Over the 11 that last under 15 s here it is 0.97; against the benchmark’s mean over all its runs, 0.98. Against the benchmark’s PhysX runs it is 0.97. Run by run, 212 of the 300 runs here end within 1 s of the same run in the benchmark’s MuJoCo (Pearson 0.95); its own PhysX runs do so 169 times (Pearson 0.91). The widest gap is wty_cpp: 71.0 s here, 42.3 s in the benchmark’s MuJoCo and 32.8 s in its PhysX, not yet explained.

Verdicts

Bars: mean survival at each stage, out of 90 s (the first, darker bar is the authors' config where one was read). Open a row for the stage table, what reading the adapter found, and the values that differ.

PolicyStops itStagesVerdictBench (s)Here (s)

rl_gymPort valuesThe port's values cost 31 s against the authors' config (90 to 59.3 s).25.321.2

Stage	Mean survival (s)	Completed
Authors' config	90.0	10/10
Port's values	59.3	4/10
Random arm walk	50.5	2/10
Plus punches	21.2	0/10
Port, full benchmark	21.2	0/10
Port, arms still, no punches	59.3	4/10

Run	0	1	2	3	4	5	6	7	8	9
Here	31	12	36	18	41	11	11	13	26	14
Benchmark, MuJoCo	10	28	52	8	56	24	13	26	21	16
Benchmark, PhysX	31	41	51	38	56	16	33	42	43	56

Read from policies/rl_gym/policy.cpp: 47 observations (base_ang_vel, projected_gravity, velocity_commands, joint_pos_rel, joint_vel_rel, last_action, gait_phase), history 1, 12 actions, owned() = 15.

Upstream config: configs/g1.yaml (unitree_rl_gym).

The policy's contract against the upstream config:
phase clock: contract runs -1 policy steps (-20 ms, -9 deg of a 0.8 s gait) ahead of training (boundary A deviation)
unlisted pose: 6 of 17 joint(s) differ; largest left_elbow_joint contract 0.6 against 0
unlisted kp: 17 of 17 joint(s) differ; largest left_elbow_joint contract 14.25 against 500
unlisted kd: 17 of 17 joint(s) differ; largest left_elbow_joint contract 0.9072 against 10
Port values against the policy's contract: no differences

clobotArm handoffStands and tracks in its own setup (90 s); the harness taking the arms costs 88 s.1.91.9

Stage	Mean survival (s)	Completed
Authors' config	90.0	10/10
Port's values	90.0	10/10
Harness holds arms	2.4	0/10
Random arm walk	1.9	0/10
Plus punches	1.9	0/10
Port, full benchmark	1.9	0/10
Port, arms still, no punches	2.4	0/10

Run	0	1	2	3	4	5	6	7	8	9
Here	2	3	2	2	1	2	2	2	2	2
Benchmark, MuJoCo	2	3	2	2	1	2	2	2	2	2
Benchmark, PhysX	2	3	2	1	2	2	2	1	2	2

Read from policies/clobot/policy.cpp: 500 observations (base_ang_vel, projected_gravity, velocity_commands, joint_pos_rel, joint_vel_rel, last_action, gait_phase_legs), history 5, 29 actions, owned() = 15.

What reading the adapter found

actions are clipped to +-5 before scaling

Upstream config: env.yaml (Isaac Lab training run).

The policy's contract against the upstream config:
scale: 29 of 29 joints differ; largest relative +0.10% (left_hip_roll_joint)
  left_hip_roll_joint and 3 more: contract 0.351, the upstream config 0.3506614664, diff +0.0003385 (+0.10%)
  left_shoulder_pitch_joint and 15 more: contract 0.439, the upstream config 0.4385773139, diff +0.0004227 (+0.10%)
  right_hip_yaw_joint and 4 more: contract 0.548, the upstream config 0.5475464652, diff +0.0004535 (+0.08%)
  left_wrist_pitch_joint and 3 more: contract 0.0745, the upstream config 0.07450087033, diff -8.703e-07 (-0.00%)
kp: 29 of 29 joints differ; largest relative +0.35% (right_shoulder_pitch_joint)
  right_shoulder_pitch_joint and 9 more: contract 14.3, the upstream config 14.2506231, diff +0.04938 (+0.35%)
  left_wrist_pitch_joint and 3 more: contract 16.8, the upstream config 16.77832748, diff +0.02167 (+0.13%)
  left_hip_yaw_joint and 4 more: contract 40.2, the upstream config 40.17923847, diff +0.02076 (+0.05%)
  right_ankle_roll_joint and 5 more: contract 28.5, the upstream config 28.5012462, diff -0.001246 (-0.00%)
  right_knee_joint and 3 more: contract 99.1, the upstream config 99.09842778, diff +0.001572 (+0.00%)
kd: 29 of 29 joints differ; largest relative -0.25% (waist_roll_joint)
  waist_roll_joint and 5 more: contract 1.81, the upstream config 1.814445687, diff -0.004446 (-0.25%)
  left_wrist_pitch_joint and 3 more: contract 1.07, the upstream config 1.068141502, diff +0.001858 (+0.17%)
  left_hip_yaw_joint and 4 more: contract 2.56, the upstream config 2.557889765, diff +0.00211 (+0.08%)
  left_shoulder_roll_joint and 9 more: contract 0.907, the upstream config 0.9072228433, diff -0.0002228 (-0.02%)
  right_knee_joint and 3 more: contract 6.31, the upstream config 6.308801853, diff +0.001198 (+0.02%)
Port values against the policy's contract:
kp: 14 of 29 joints differ; largest relative -0.35% (left_shoulder_pitch_joint)
  left_shoulder_pitch_joint and 9 more: port 14.2506, the policy's own contract 14.3, diff -0.04938 (-0.35%)
  left_wrist_pitch_joint and 3 more: port 16.7783, the policy's own contract 16.8, diff -0.02167 (-0.13%)
kd: 14 of 29 joints differ; largest relative -0.17% (left_wrist_yaw_joint)
  left_wrist_yaw_joint and 3 more: port 1.06814, the policy's own contract 1.07, diff -0.001858 (-0.17%)
  left_shoulder_pitch_joint and 9 more: port 0.907223, the policy's own contract 0.907, diff +0.0002228 (+0.02%)
port: harness drives 14 joint(s) (hold), policy sees them as real: left_shoulder_pitch_joint, right_shoulder_pitch_joint, left_shoulder_roll_joint, right_shoulder_roll_joint, left_shoulder_yaw_joint, right_shoulder_yaw_joint ...

legged_rl_labArm handoffLasts 87 s in its own setup (9/10 complete); the harness taking the arms costs 69 s.3.74.8

Stage	Mean survival (s)	Completed
Port's values	87.3	9/10
Harness holds arms	18.8	0/10
Random arm walk	4.2	0/10
Plus punches	4.8	0/10
Port, full benchmark	4.8	0/10
Port, arms still, no punches	18.8	0/10

Run	0	1	2	3	4	5	6	7	8	9
Here	4	2	3	4	4	4	3	5	9	9
Benchmark, MuJoCo	3	3	3	2	2	2	4	3	3	11
Benchmark, PhysX	4	3	4	3	4	4	3	2	2	8

Read from policies/legged_rl_lab/policy.cpp: 96 observations (base_ang_vel, projected_gravity, velocity_commands, joint_pos_rel, joint_vel_rel, last_action), history 1, 29 actions, owned() = 15.

What reading the adapter found

the action gain at the first step is 0, 0.25 after 2 s: the port ramps its actions in

No upstream config read.

Port values against the policy's contract:
kp: 14 of 29 joints differ; largest relative -64.37% (left_shoulder_pitch_joint)
  left_shoulder_pitch_joint and 9 more: port 14.2506, the policy's own contract 40, diff -25.75 (-64.37%)
  left_wrist_pitch_joint and 3 more: port 16.7783, the policy's own contract 40, diff -23.22 (-58.05%)
kd: 14 of 29 joints differ; largest relative -9.28% (left_shoulder_pitch_joint)
  left_shoulder_pitch_joint and 9 more: port 0.907223, the policy's own contract 1, diff -0.09278 (-9.28%)
  left_wrist_pitch_joint and 3 more: port 1.06814, the policy's own contract 1, diff +0.06814 (+6.81%)
port: harness drives 14 joint(s) (hold), policy sees them as real: left_shoulder_pitch_joint, right_shoulder_pitch_joint, left_shoulder_roll_joint, right_shoulder_roll_joint, left_shoulder_yaw_joint, right_shoulder_yaw_joint ...

asapArm walkStands and tracks in its own setup (90 s); the random arm walk costs 37 s.48.749.5

Stage	Mean survival (s)	Completed
Port's values	90.0	10/10
Random arm walk	52.6	3/10
Plus punches	49.5	1/10
Port, full benchmark	49.5	1/10
Port, arms still, no punches	90.0	10/10

Run	0	1	2	3	4	5	6	7	8	9
Here	23	56	26	71	42	48	26	76	37	90
Benchmark, MuJoCo	23	54	26	71	46	48	26	73	40	81
Benchmark, PhysX	21	53	26	71	22	15	31	90	35	90

Read from policies/asap/policy.cpp: 500 observations (last_action, base_ang_vel, velocity_commands*, velocity_commands_2*, command_gate, gait_phase*, joint_pos_rel, joint_vel_rel, projected_gravity, joint_target_rel*, gait_phase_2*, constant, constant_2, constant_3), laid out as 62 [term, lag] chunks, 12 actions, owned() = 15.

What reading the adapter found

the port observes the harness's targets for motor(s) [15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28] (joint_target_rel)
the port observes motor(s) [12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28] that no action drives (held by the port or the harness): read after the policy's joints
21 observation element(s) are constants the port feeds: [16]=1.56, [19]=1, [79]=1.56, [80]=1.56, [81]=1.56, [82]=1.56, [83]=0, [84]=0 ...
the port passes its command only once the waypoint is farther than 0.08 m or 0.1 rad off, until it is within 0.04 m and 0.05 rad (a walk latch on the task)
the observation is one frame of 116 elements laid out 62 times by term and age (history.chunks, lags up to 4)

No upstream config read.

Port values against the policy's contract: no differences

dm_marchArm walkLasts 81 s in its own setup (9/10 complete); the random arm walk costs 62 s.4.34.3

Stage	Mean survival (s)	Completed
Authors' config	90.0	10/10
Port's values	81.2	9/10
Harness holds arms	90.0	10/10
Random arm walk	28.0	0/10
Plus punches	21.5	0/10
Port, full benchmark	4.3	0/10
Port, arms still, no punches	19.6	2/10

Run	0	1	2	3	4	5	6	7	8	9
Here	1	1	2	2	2	2	27	2	2	3
Benchmark, MuJoCo	1	1	2	2	2	2	27	2	2	3
Benchmark, PhysX	1	1	2	1	2	2	2	1	2	11

With the arms held still and no punches, the port's own view of the arms costs 70 s (90.0 to 19.6 s).

Read from policies/dm_march/policy.cpp: 490 observations (base_ang_vel, projected_gravity, velocity_commands, joint_pos_rel, joint_vel_rel, last_action, gait_phase), history 5, 29 actions, owned() = 15.

Upstream config: env.yaml (Isaac Lab training run).

The policy's contract against the upstream config: no differences
Port values against the policy's contract:
kp: 14 of 29 joints differ; largest relative -64.37% (left_shoulder_pitch_joint)
  left_shoulder_pitch_joint and 9 more: port 14.2506, the policy's own contract 40, diff -25.75 (-64.37%)
  left_wrist_pitch_joint and 3 more: port 16.7783, the policy's own contract 40, diff -23.22 (-58.05%)
kd: 14 of 29 joints differ; largest relative -9.28% (left_shoulder_pitch_joint)
  left_shoulder_pitch_joint and 9 more: port 0.907223, the policy's own contract 1, diff -0.09278 (-9.28%)
  left_wrist_pitch_joint and 3 more: port 1.06814, the policy's own contract 1, diff +0.06814 (+6.81%)
port: harness drives 14 joint(s) (hold), policy sees them as echo_action: left_shoulder_pitch_joint, right_shoulder_pitch_joint, left_shoulder_roll_joint, right_shoulder_roll_joint, left_shoulder_yaw_joint, right_shoulder_yaw_joint ...

handoffArm walkStands and tracks in its own setup (90 s); the random arm walk costs 61 s.23.723.9

Stage	Mean survival (s)	Completed
Port's values	90.0	10/10
Harness holds arms	90.0	10/10
Random arm walk	29.2	0/10
Plus punches	23.9	0/10
Port, full benchmark	23.9	0/10
Port, arms still, no punches	90.0	10/10

Run	0	1	2	3	4	5	6	7	8	9
Here	13	16	25	36	22	57	17	17	24	14
Benchmark, MuJoCo	12	17	25	32	19	66	19	17	16	14
Benchmark, PhysX	14	25	29	21	21	48	20	17	24	13

Read from policies/handoff/policy.cpp: 1272 observations (velocity_commands*, velocity_commands_2*, gait_phase_legs*, base_ang_vel, gravity_euler, joint_pos_rel, joint_vel_rel*, joint_vel_rel_2*, joint_vel_rel_3*, last_action, constant, constant_2, constant_3), laid out as 168 [term, lag] chunks, 29 actions, owned() = 15.

What reading the adapter found

obs [17, 18] are roll and pitch computed from gravity (atan2, asin), not gravity itself
the port zeroes its command while its norm is below 0.1
the two-leg clock reads phase zero while the command's norm is below 0.1, running on underneath
132 observation element(s) are constants the port feeds: [2]=0.78, [4]=0.04, [5]=0.230447, [6]=-0.0784201, [7]=0.04, [8]=-0.230437, [9]=-0.0784201, [52]=0 ...
joint_vel_rel of policy joints [4, 5, 10, 11] is not observed
the observation is one frame of 227 elements laid out 168 times by term and age (history.chunks, lags up to 10)

No upstream config read.

Port values against the policy's contract:
kp: 14 of 29 joints differ; largest relative +0.00% (left_wrist_yaw_joint)
  left_wrist_yaw_joint and 3 more: port 16.7783, the policy's own contract 16.7783, diff +2.748e-05 (+0.00%)
  left_elbow_joint and 9 more: port 14.2506, the policy's own contract 14.2506, diff +2.31e-05 (+0.00%)
kd: 14 of 29 joints differ; largest relative +0.00% (left_shoulder_pitch_joint)
  left_shoulder_pitch_joint and 9 more: port 0.907223, the policy's own contract 0.90722, diff +2.843e-06 (+0.00%)
  left_wrist_yaw_joint and 3 more: port 1.06814, the policy's own contract 1.06814, diff +1.502e-06 (+0.00%)
port: harness drives 14 joint(s) (hold), policy sees them as real: left_shoulder_pitch_joint, left_shoulder_roll_joint, left_shoulder_yaw_joint, left_elbow_joint, left_wrist_roll_joint, left_wrist_pitch_joint ...

holosomaArm walkStands and tracks in its own setup (90 s); the random arm walk costs 85 s.6.66.2

Stage	Mean survival (s)	Completed
Port's values	90.0	10/10
Harness holds arms	90.0	10/10
Random arm walk	4.5	0/10
Plus punches	6.2	0/10
Port, full benchmark	6.2	0/10
Port, arms still, no punches	90.0	10/10

Run	0	1	2	3	4	5	6	7	8	9
Here	9	4	15	8	6	7	4	4	4	2
Benchmark, MuJoCo	13	4	15	8	5	7	4	4	3	2
Benchmark, PhysX	11	4	15	11	8	8	2	4	3	3

Read from policies/holosoma/policy.cpp: 100 observations (last_action, base_ang_vel, velocity_commands*, gait_phase_legs*, joint_pos_rel, joint_vel_rel, projected_gravity, gait_phase_legs_2*), history 1, 29 actions, owned() = 15.

What reading the adapter found

the two-leg clock holds at [0.5, 0.5] (cycles) while |(vx, vy)| < 0.01 and |wz| < 0.01, and restarts at [0.0, 0.5]

No upstream config read.

Port values against the policy's contract: no differences
port: harness drives 14 joint(s) (hold), policy sees them as real: left_shoulder_pitch_joint, left_shoulder_roll_joint, left_shoulder_yaw_joint, left_elbow_joint, left_wrist_roll_joint, left_wrist_pitch_joint ...

huruArm walkStands and tracks in its own setup (90 s); the random arm walk costs 81 s.7.36.3

Stage	Mean survival (s)	Completed
Port's values	90.0	10/10
Harness holds arms	90.0	10/10
Random arm walk	8.6	0/10
Plus punches	6.3	0/10
Port, full benchmark	6.3	0/10
Port, arms still, no punches	90.0	10/10

Run	0	1	2	3	4	5	6	7	8	9
Here	6	8	4	7	10	7	6	4	5	7
Benchmark, MuJoCo	7	8	4	7	10	9	6	6	5	12
Benchmark, PhysX	3	7	17	4	11	9	5	12	5	5

Read from policies/huru/policy.cpp: 99 observations (base_lin_vel, base_ang_vel, projected_gravity, joint_pos_rel, joint_vel_rel, last_action, velocity_commands), history 1, 29 actions, owned() = 15.

No upstream config read.

Port values against the policy's contract:
kp: 14 of 29 joints differ; largest relative -0.00% (left_shoulder_pitch_joint)
  left_shoulder_pitch_joint and 9 more: port 14.2506, the policy's own contract 14.251, diff -0.0003769 (-0.00%)
  left_wrist_yaw_joint and 3 more: port 16.7783, the policy's own contract 16.778, diff +0.0003275 (+0.00%)
kd: 14 of 29 joints differ; largest relative +0.02% (left_shoulder_pitch_joint)
  left_shoulder_pitch_joint and 9 more: port 0.907223, the policy's own contract 0.907, diff +0.0002228 (+0.02%)
  left_wrist_yaw_joint and 3 more: port 1.06814, the policy's own contract 1.068, diff +0.0001415 (+0.01%)
port: harness drives 14 joint(s) (hold), policy sees them as real: left_shoulder_pitch_joint, left_shoulder_roll_joint, left_shoulder_yaw_joint, left_elbow_joint, left_wrist_roll_joint, left_wrist_pitch_joint ...

rl_labArm walkStands and tracks in its own setup (90 s); the random arm walk costs 83 s.34.833.2

Stage	Mean survival (s)	Completed
Port's values	90.0	10/10
Harness holds arms	90.0	10/10
Random arm walk	7.3	0/10
Plus punches	7.8	0/10
Port, full benchmark	33.2	0/10
Port, arms still, no punches	90.0	10/10

Run	0	1	2	3	4	5	6	7	8	9
Here	18	29	43	19	31	14	31	48	66	33
Benchmark, MuJoCo	21	29	43	18	33	13	32	47	88	24
Benchmark, PhysX	18	12	43	17	12	24	31	46	19	32

The port survives 25 s longer than the clean contract under the full benchmark: it hides the arm motion from the policy.

Read from policies/rl_lab/policy.cpp: 480 observations (base_ang_vel, projected_gravity, velocity_commands, joint_pos_rel, joint_vel_rel, last_action), history 5, 29 actions, owned() = 15.

What reading the adapter found

actions are clipped to +-10 before scaling

No upstream config read.

Port values against the policy's contract:
kp: 14 of 29 joints differ; largest relative -64.37% (left_shoulder_pitch_joint)
  left_shoulder_pitch_joint and 9 more: port 14.2506, the policy's own contract 40, diff -25.75 (-64.37%)
  left_wrist_pitch_joint and 3 more: port 16.7783, the policy's own contract 40, diff -23.22 (-58.05%)
kd: 14 of 29 joints differ; largest relative -90.93% (left_shoulder_pitch_joint)
  left_shoulder_pitch_joint and 9 more: port 0.907223, the policy's own contract 10, diff -9.093 (-90.93%)
  left_wrist_pitch_joint and 3 more: port 1.06814, the policy's own contract 10, diff -8.932 (-89.32%)
port: harness drives 14 joint(s) (hold), policy sees them as echo_action: left_shoulder_pitch_joint, right_shoulder_pitch_joint, left_shoulder_roll_joint, right_shoulder_roll_joint, left_shoulder_yaw_joint, right_shoulder_yaw_joint ...

rl_mjlabArm walkStands and tracks in its own setup (90 s); the random arm walk costs 85 s.4.94.9

Stage	Mean survival (s)	Completed
Port's values	90.0	10/10
Harness holds arms	90.0	10/10
Random arm walk	5.0	0/10
Plus punches	4.9	0/10
Port, full benchmark	4.9	0/10
Port, arms still, no punches	90.0	10/10

Run	0	1	2	3	4	5	6	7	8	9
Here	4	6	5	3	7	4	4	3	10	3
Benchmark, MuJoCo	4	5	4	3	7	4	4	3	11	3
Benchmark, PhysX	4	6	5	5	6	4	5	3	4	2

Read from policies/rl_mjlab/policy.cpp: 98 observations (base_ang_vel, projected_gravity, velocity_commands, gait_phase, joint_pos_rel, joint_vel_rel, last_action), history 1, 29 actions, owned() = 15.

No upstream config read.

Port values against the policy's contract:
kp: 14 of 29 joints differ; largest relative -0.35% (left_shoulder_pitch_joint)
  left_shoulder_pitch_joint and 9 more: port 14.2506, the policy's own contract 14.3, diff -0.04938 (-0.35%)
  left_wrist_yaw_joint and 3 more: port 16.7783, the policy's own contract 16.8, diff -0.02167 (-0.13%)
kd: 14 of 29 joints differ; largest relative -2.90% (left_wrist_yaw_joint)
  left_wrist_yaw_joint and 3 more: port 1.06814, the policy's own contract 1.1, diff -0.03186 (-2.90%)
  left_elbow_joint and 9 more: port 0.907223, the policy's own contract 0.9, diff +0.007223 (+0.80%)
port: harness drives 14 joint(s) (hold), policy sees them as real: left_shoulder_pitch_joint, left_shoulder_roll_joint, left_shoulder_yaw_joint, left_elbow_joint, left_wrist_roll_joint, left_wrist_pitch_joint ...

robomimicArm walkStands and tracks in its own setup (90 s); the random arm walk costs 85 s.54.549.9

Stage	Mean survival (s)	Completed
Port's values	90.0	10/10
Harness holds arms	90.0	10/10
Random arm walk	4.5	0/10
Plus punches	4.2	0/10
Port, full benchmark	49.9	1/10
Port, arms still, no punches	90.0	10/10

Run	0	1	2	3	4	5	6	7	8	9
Here	42	48	82	31	56	23	46	56	90	25
Benchmark, MuJoCo	39	35	90	35	41	25	46	56	90	90
Benchmark, PhysX	38	47	81	30	41	26	46	46	90	24

The port survives 46 s longer than the clean contract under the full benchmark: it hides the arm motion from the policy.

Read from policies/robomimic/policy.cpp: 96 observations (base_ang_vel, projected_gravity, velocity_commands, joint_pos_rel, joint_vel_rel, last_action), history 1, 29 actions, owned() = 15.

What reading the adapter found

actions are clipped to +-100 before scaling

No upstream config read.

Port values against the policy's contract: no differences
port: harness drives 14 joint(s) (hold), policy sees them as default: left_shoulder_pitch_joint, right_shoulder_pitch_joint, left_shoulder_roll_joint, right_shoulder_roll_joint, left_shoulder_yaw_joint, right_shoulder_yaw_joint ...

wty_cppArm walkStands and tracks in its own setup (90 s); the random arm walk costs 68 s.42.371.0

Stage	Mean survival (s)	Completed
Authors' config	90.0	10/10
Port's values	90.0	10/10
Harness holds arms	90.0	10/10
Random arm walk	21.9	0/10
Plus punches	21.6	0/10
Port, full benchmark	71.0	3/10
Port, arms still, no punches	90.0	10/10

Run	0	1	2	3	4	5	6	7	8	9
Here	90	61	90	71	46	86	61	61	90	56
Benchmark, MuJoCo	90	36	22	6	47	58	7	6	62	90
Benchmark, PhysX	81	11	22	2	46	56	25	41	21	24

The port survives 49 s longer than the clean contract under the full benchmark: it hides the arm motion from the policy.

Read from policies/wty_cpp/policy.cpp: 480 observations (base_ang_vel, projected_gravity, velocity_commands, joint_pos_rel, joint_vel_rel, last_action), history 5, 29 actions, owned() = 15.

What reading the adapter found

actions are clipped to +-10 before scaling

Upstream config: deploy.yaml (Unitree).

The policy's contract against the upstream config: no differences
Port values against the policy's contract:
kp: 14 of 29 joints differ; largest relative -71.50% (left_shoulder_pitch_joint)
  left_shoulder_pitch_joint and 5 more: port 14.2506, the policy's own contract 50, diff -35.75 (-71.50%)
  left_wrist_yaw_joint and 3 more: port 16.7783, the policy's own contract 16.7783, diff +2.748e-05 (+0.00%)
  left_wrist_roll_joint and 3 more: port 14.2506, the policy's own contract 14.2506, diff +2.31e-05 (+0.00%)
kd: 14 of 29 joints differ; largest relative +0.00% (left_wrist_yaw_joint)
  left_wrist_yaw_joint and 3 more: port 1.06814, the policy's own contract 1.0681, diff +4.15e-05 (+0.00%)
  left_elbow_joint and 9 more: port 0.907223, the policy's own contract 0.9072, diff +2.284e-05 (+0.00%)
port: harness drives 14 joint(s) (hold), policy sees them as echo_action: left_shoulder_pitch_joint, left_shoulder_roll_joint, left_shoulder_yaw_joint, left_elbow_joint, left_wrist_roll_joint, left_wrist_pitch_joint ...

decoupled_wbch074_p000PunchesStands and tracks in its own setup (90 s); punching costs 11 s.85.778.8

Stage	Mean survival (s)	Completed
Port's values	90.0	10/10
Random arm walk	90.0	10/10
Plus punches	78.8	5/10
Port, full benchmark	78.8	5/10
Port, arms still, no punches	90.0	10/10

Run	0	1	2	3	4	5	6	7	8	9
Here	90	61	90	90	41	86	71	90	90	81
Benchmark, MuJoCo	90	90	90	71	90	86	71	90	90	90
Benchmark, PhysX	90	60	90	90	46	90	71	90	90	90

Read from policies/decoupled_wbc/policy.cpp (variant decoupled_wbc_h074_p000): 116 observations (base_ang_vel, projected_gravity, velocity_commands, constant, joint_pos_rel, joint_vel_rel, last_action), history 2, 15 actions, owned() = 15.

What reading the adapter found

4 observation element(s) are constants the port feeds: [67]=0.74, [68]=0, [69]=0, [70]=0

No upstream config read.

Port values against the policy's contract: no differences

falconPunchesStands and tracks in its own setup (90 s); punching costs 35 s.31.846.1

Stage	Mean survival (s)	Completed
Port's values	90.0	10/10
Random arm walk	81.3	8/10
Plus punches	46.1	0/10
Port, full benchmark	46.1	0/10
Port, arms still, no punches	90.0	10/10

Run	0	1	2	3	4	5	6	7	8	9
Here	56	51	61	26	40	31	46	26	41	82
Benchmark, MuJoCo	17	46	46	27	40	11	46	42	21	21
Benchmark, PhysX	16	26	62	36	46	66	12	50	61	66

Read from policies/falcon/policy.cpp: 102 observations (last_action, base_ang_vel, velocity_commands*, command_gate, constant, gait_phase_gated*, joint_pos_rel, constant_2, joint_vel_rel, constant_3, projected_gravity, constant_4, gait_phase_gated_2*), history 1, 15 actions, owned() = 15.

What reading the adapter found

the port makes its own command from the harness's task: the command's direction at min(2 x distance, 0.9) m/s, yaw as commanded (speeds factored assuming the port caps at limits() speed_norm 0.9)
the port passes its command only while it is nonzero and 1.5 s after a start, and feeds that as a flag (obs [21])
a gait clock of period 0.9 s that runs only while the gate is open, advancing before it is read
45 observation element(s) are constants the port feeds: [22]=0, [23]=0, [24]=0, [41]=0, [42]=0, [43]=0, [44]=0, [45]=0 ...

No upstream config read.

Port values against the policy's contract: no differences

gr00t_wbch070_p000PunchesStands and tracks in its own setup (90 s); punching costs 6 s.87.183.7

Stage	Mean survival (s)	Completed
Port's values	90.0	10/10
Random arm walk	90.0	10/10
Plus punches	83.7	8/10
Port, full benchmark	83.7	8/10
Port, arms still, no punches	90.0	10/10

Run	0	1	2	3	4	5	6	7	8	9
Here	90	61	90	90	56	90	90	90	90	90
Benchmark, MuJoCo	90	61	90	90	90	90	90	90	90	90
Benchmark, PhysX	66	90	90	90	90	90	90	90	90	90

Read from policies/gr00t_wbc/policy.cpp (variant gr00t_wbc_h070_p000): 516 observations (velocity_commands, constant, base_ang_vel, projected_gravity, joint_pos_rel, joint_vel_rel, last_action), history 6, 15 actions, owned() = 15.

What reading the adapter found

the port observes motor(s) [15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28] that no action drives (held by the port or the harness): read after the policy's joints
4 observation element(s) are constants the port feeds: [433]=0.7, [434]=0, [435]=0, [436]=0
the port runs model_walk.onnx while the command's norm exceeds 0.05, else model_balance.onnx

No upstream config read.

Port values against the policy's contract: no differences

gr00t_wbch074_p000PunchesStands and tracks in its own setup (90 s); punching costs 8 s.81.881.8

Stage	Mean survival (s)	Completed
Port's values	90.0	10/10
Random arm walk	90.0	10/10
Plus punches	81.8	7/10
Port, full benchmark	81.8	7/10
Port, arms still, no punches	90.0	10/10

Run	0	1	2	3	4	5	6	7	8	9
Here	90	61	90	71	56	90	90	90	90	90
Benchmark, MuJoCo	90	61	90	71	56	90	90	90	90	90
Benchmark, PhysX	90	61	90	90	56	90	71	90	90	90

Read from policies/gr00t_wbc/policy.cpp (variant gr00t_wbc_h074_p000): 516 observations (velocity_commands, constant, base_ang_vel, projected_gravity, joint_pos_rel, joint_vel_rel, last_action), history 6, 15 actions, owned() = 15.

What reading the adapter found

the port observes motor(s) [15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28] that no action drives (held by the port or the harness): read after the policy's joints
4 observation element(s) are constants the port feeds: [433]=0.74, [434]=0, [435]=0, [436]=0
the port runs model_walk.onnx while the command's norm exceeds 0.05, else model_balance.onnx

No upstream config read.

Port values against the policy's contract: no differences

homiePunchesStands and tracks in its own setup (90 s); punching costs 11 s.79.579.4

Stage	Mean survival (s)	Completed
Port's values	90.0	10/10
Random arm walk	90.0	10/10
Plus punches	79.4	5/10
Port, full benchmark	79.4	5/10
Port, arms still, no punches	90.0	10/10

Run	0	1	2	3	4	5	6	7	8	9
Here	90	61	90	71	56	86	71	90	90	90
Benchmark, MuJoCo	90	61	90	72	56	86	71	90	90	90
Benchmark, PhysX	90	77	53	71	57	90	47	90	61	90

Read from policies/homie/policy.cpp: 456 observations (velocity_commands, constant, base_ang_vel, projected_gravity, joint_pos_rel, constant_2, joint_vel_rel, constant_3, last_action), history 6, 12 actions, owned() = 15.

What reading the adapter found

the port feeds zeros at the first step (it builds no observation yet)
the port observes motor(s) [12] that no action drives (held by the port or the harness): read after the policy's joints
the port makes its own command from the harness's task: the command's direction at min(0.8 x distance, 0.4) m/s, yaw toward the target, facing travel when far (speeds factored assuming the port caps at limits() speed_norm 0.4)
29 observation element(s) are constants the port feeds: [383]=0.74, [403]=0, [404]=0, [405]=0, [406]=0, [407]=0, [408]=0, [409]=0 ...
actions are clipped to +-100 before scaling

No upstream config read.

Port values against the policy's contract: no differences

openwbtPunchesStands and tracks in its own setup (90 s); punching costs 39 s.33.134.1

Stage	Mean survival (s)	Completed
Port's values	90.0	10/10
Random arm walk	73.3	6/10
Plus punches	34.1	0/10
Port, full benchmark	34.1	0/10
Port, arms still, no punches	90.0	10/10

Run	0	1	2	3	4	5	6	7	8	9
Here	12	26	36	41	56	26	26	54	41	23
Benchmark, MuJoCo	12	36	51	21	46	19	26	54	41	24
Benchmark, PhysX	12	54	52	43	46	57	26	42	41	76

Read from policies/openwbt/policy.cpp: 47 observations (velocity_commands, projected_gravity, base_ang_vel, joint_pos_rel, joint_vel_rel, last_action, gait_phase_feet), history 1, 12 actions, owned() = 15.

What reading the adapter found

clock inputs per foot (walk-these-ways): 1.5 Hz, stance ratio 0.6, offsets [0.5, 0.0], from index 0.3; held at 0.3 while the command is zero
actions are clipped to +-100 before scaling

No upstream config read.

Port values against the policy's contract: no differences

schoiPunchesStands and tracks in its own setup (90 s); punching costs 55 s.37.134.8

Stage	Mean survival (s)	Completed
Authors' config	90.0	10/10
Port's values	90.0	10/10
Random arm walk	90.0	10/10
Plus punches	34.8	1/10
Port, full benchmark	34.8	1/10
Port, arms still, no punches	90.0	10/10

Run	0	1	2	3	4	5	6	7	8	9
Here	31	21	31	44	16	16	26	26	48	90
Benchmark, MuJoCo	31	11	31	46	46	17	26	26	48	90
Benchmark, PhysX	36	41	33	7	23	16	26	58	29	57

Read from policies/schoi/policy.cpp: 47 observations (base_ang_vel, projected_gravity, velocity_commands, joint_pos_rel, joint_vel_rel, last_action, gait_phase), history 1, 12 actions, owned() = 12.

Upstream config: g1_sim2real.yaml (hand contract).

The policy's contract against the upstream config:
unlisted pose: 8 of 17 joint(s) differ; largest left_elbow_joint contract 0.6 against 0.97
unlisted kp: 14 of 17 joint(s) differ; largest left_shoulder_pitch_joint contract 14.25 against 100
unlisted kd: 14 of 17 joint(s) differ; largest left_elbow_joint contract 0.9072 against 2
Port values against the policy's contract: no differences

zealotPunchesStands and tracks in its own setup (90 s); punching costs 17 s.79.573.4

Stage	Mean survival (s)	Completed
Port's values	90.0	10/10
Random arm walk	90.0	10/10
Plus punches	73.4	5/10
Port, full benchmark	73.4	5/10
Port, arms still, no punches	90.0	10/10

Run	0	1	2	3	4	5	6	7	8	9
Here	90	61	51	71	40	90	90	61	90	90
Benchmark, MuJoCo	90	60	90	87	47	90	90	61	90	90
Benchmark, PhysX	32	52	90	46	90	56	47	61	90	90

Read from policies/zealot/policy.cpp: 265 observations (last_action, velocity_commands, constant, joint_pos_rel, joint_vel_diff, projected_gravity, gait_phase_speed, base_ang_vel, constant_2), history 5, 12 actions, owned() = 12.

What reading the adapter found

the port observes the action from 2 steps back
a gait clock whose period follows the command speed (norm3): 0.8 s at 0.1, 0.4 s at 0.74, 0.4 s at 2; it holds below 0.1
6 observation element(s) are constants the port feeds: [227]=0, [260]=0, [261]=0, [262]=0, [263]=0, [264]=0
the port steers from the harness's task, not its velocity command: a waypoint follower with walk_p 1.5, walk_speed 0.5, yaw_p 1.2, yaw_rate_abs 0.6, vy_abs 0.3, face_near_m 0.4, face_far_m 1.5

No upstream config read.

Port values against the policy's contract: no differences

dm_agilePolicyFalls under its authors' own config (2.2 s of 90): the score belongs to the policy, not the port.1.81.8

Stage	Mean survival (s)	Completed
Authors' config	2.2	0/10
Port's values	2.1	0/10
Harness holds arms	3.0	0/10
Random arm walk	2.9	0/10
Plus punches	2.0	0/10
Port, full benchmark	1.8	0/10
Port, arms still, no punches	1.8	0/10

Run	0	1	2	3	4	5	6	7	8	9
Here	2	2	2	1	2	2	3	2	2	2
Benchmark, MuJoCo	2	2	2	1	2	2	3	2	2	2
Benchmark, PhysX	2	2	2	1	2	2	2	2	2	2

Read from policies/dm_agile/policy.cpp: 480 observations (base_ang_vel, projected_gravity, velocity_commands, joint_pos_rel, joint_vel_rel, last_action), history 5, 29 actions, owned() = 15.

What reading the adapter found

actions are clipped to +-10 before scaling

Upstream config: env.yaml (Isaac Lab training run).

The policy's contract against the upstream config: no differences
Port values against the policy's contract:
kp: 14 of 29 joints differ; largest relative -64.37% (left_shoulder_pitch_joint)
  left_shoulder_pitch_joint and 9 more: port 14.2506, the policy's own contract 40, diff -25.75 (-64.37%)
  left_wrist_pitch_joint and 3 more: port 16.7783, the policy's own contract 40, diff -23.22 (-58.05%)
kd: 14 of 29 joints differ; largest relative -9.28% (left_shoulder_pitch_joint)
  left_shoulder_pitch_joint and 9 more: port 0.907223, the policy's own contract 1, diff -0.09278 (-9.28%)
  left_wrist_pitch_joint and 3 more: port 1.06814, the policy's own contract 1, diff +0.06814 (+6.81%)
port: harness drives 14 joint(s) (hold), policy sees them as echo_action: left_shoulder_pitch_joint, right_shoulder_pitch_joint, left_shoulder_roll_joint, right_shoulder_roll_joint, left_shoulder_yaw_joint, right_shoulder_yaw_joint ...

g1_gymPolicyFalls under its authors' own config (4.5 s of 90): the score belongs to the policy, not the port.2.73.3

Stage	Mean survival (s)	Completed
Authors' config	4.5	0/10
Port's values	6.6	0/10
Harness holds arms	3.2	0/10
Random arm walk	3.0	0/10
Plus punches	3.3	0/10
Port, full benchmark	3.3	0/10
Port, arms still, no punches	3.2	0/10

Run	0	1	2	3	4	5	6	7	8	9
Here	4	5	1	7	2	2	2	5	2	3
Benchmark, MuJoCo	2	4	3	4	2	2	3	2	3	2
Benchmark, PhysX	2	2	1	2	1	1	2	3	3	5

Read from policies/g1_gym/policy.cpp: 90 observations (base_ang_vel, projected_gravity, velocity_commands, joint_pos_rel, joint_vel_rel, last_action), history 1, 27 actions, owned() = 15.

Upstream config: g1_full.yaml (hand contract).

The policy's contract against the upstream config:
kp: 20 of 27 joints differ; largest relative -71.50% (left_shoulder_pitch_joint)
  left_shoulder_pitch_joint and 7 more: contract 14.2506, the upstream config 50, diff -35.75 (-71.50%)
  wrist_roll_joint (both sides): contract 14.2506, the upstream config 30, diff -15.75 (-52.50%)
  left_ankle_roll_joint and 3 more: contract 20, the upstream config 40, diff -20 (-50.00%)
  left_wrist_pitch_joint and 3 more: contract 16.7783, the upstream config 30, diff -13.22 (-44.07%)
  knee_joint (both sides): contract 200, the upstream config 150, diff +50 (+33.33%)
kd: 27 of 27 joints differ; largest relative +100.00% (waist_yaw_joint)
  waist_yaw_joint: contract 4, the upstream config 2, diff +2 (+100.00%)
  ankle_roll_joint (both sides): contract 0.1, the upstream config 2, diff -1.9 (-95.00%)
  ankle_pitch_joint (both sides): contract 0.2, the upstream config 2, diff -1.8 (-90.00%)
  left_shoulder_pitch_joint and 9 more: contract 0.907223, the upstream config 2, diff -1.093 (-54.64%)
  left_wrist_yaw_joint and 3 more: contract 1.06814, the upstream config 2, diff -0.9319 (-46.59%)
  left_hip_roll_joint and 5 more: contract 2.5, the upstream config 2, diff +0.5 (+25.00%)
  knee_joint (both sides): contract 5, the upstream config 4, diff +1 (+25.00%)
unlisted pose: same on 2 joint(s)
unlisted kp: same on 2 joint(s)
unlisted kd: same on 2 joint(s)
Port values against the policy's contract: no differences
port: harness drives 14 joint(s) (hold), policy sees them as real: left_shoulder_pitch_joint, left_shoulder_roll_joint, left_shoulder_yaw_joint, left_elbow_joint, left_wrist_roll_joint, left_wrist_pitch_joint ...

nanog1PolicyFalls under its authors' own config (2.4 s of 90): the score belongs to the policy, not the port.2.12.2

Stage	Mean survival (s)	Completed
Authors' config	2.4	0/10
Port's values	3.1	0/10
Harness holds arms	3.4	0/10
Random arm walk	1.9	0/10
Plus punches	1.8	0/10
Port, full benchmark	2.2	0/10
Port, arms still, no punches	2.9	0/10

Run	0	1	2	3	4	5	6	7	8	9
Here	2	2	3	2	2	2	1	3	2	2
Benchmark, MuJoCo	2	2	3	2	4	2	2	2	1	2
Benchmark, PhysX	3	2	3	4	3	2	2	3	1	2

Read from policies/nanog1/policy.cpp: 98 observations (base_ang_vel, projected_gravity, velocity_commands, joint_pos_rel, joint_vel_rel, last_action*, constant, gait_phase), history 1, 29 actions, owned() = 15.

What reading the adapter found

17 observation element(s) are constants the port feeds: [79]=0, [80]=0, [81]=0, [82]=0, [83]=0, [84]=0, [85]=0, [86]=0 ...
actions are clipped to +-1 before scaling

Upstream config: deploy_g1.py (authors' deploy).

The policy's contract against the upstream config:
kp: 14 of 29 joints differ; largest relative +738.92% (left_wrist_yaw_joint)
  left_wrist_yaw_joint and 3 more: contract 16.7783, the upstream config 2, diff +14.78 (+738.92%)
  wrist_roll_joint (both sides): contract 14.2506, the upstream config 2, diff +12.25 (+612.53%)
  left_elbow_joint and 7 more: contract 14.2506, the upstream config 75, diff -60.75 (-81.00%)
kd: 26 of 29 joints differ; largest relative +434.07% (right_wrist_pitch_joint)
  right_wrist_pitch_joint and 3 more: contract 1.06814, the upstream config 0.2, diff +0.8681 (+434.07%)
  wrist_roll_joint (both sides): contract 0.907223, the upstream config 0.2, diff +0.7072 (+353.61%)
  left_hip_pitch_joint and 5 more: contract 4, the upstream config 2, diff +2 (+100.00%)
  left_elbow_joint and 7 more: contract 0.907223, the upstream config 2, diff -1.093 (-54.64%)
  ankle_pitch_joint (both sides): contract 3, the upstream config 2, diff +1 (+50.00%)
  knee_joint (both sides): contract 6, the upstream config 4, diff +2 (+50.00%)
  ankle_roll_joint (both sides): contract 2.2, the upstream config 2, diff +0.2 (+10.00%)
Port values against the policy's contract: no differences
port: harness drives 3 joint(s) (hold), policy sees them as real: waist_yaw_joint, waist_roll_joint, waist_pitch_joint
port: harness drives 14 joint(s) (hold), policy sees them as real: left_shoulder_pitch_joint, left_shoulder_roll_joint, left_shoulder_yaw_joint, left_elbow_joint, left_wrist_roll_joint, left_wrist_pitch_joint ...

stepdownPolicyFalls under its authors' own config (1.4 s of 90): the score belongs to the policy, not the port.1.81.6

Stage	Mean survival (s)	Completed
Authors' config	1.4	0/10
Port's values	1.5	0/10
Random arm walk	1.5	0/10
Plus punches	1.6	0/10
Port, full benchmark	1.6	0/10
Port, arms still, no punches	1.5	0/10

Run	0	1	2	3	4	5	6	7	8	9
Here	2	1	2	3	2	1	1	1	1	1
Benchmark, MuJoCo	2	1	3	2	2	2	2	1	1	2
Benchmark, PhysX	2	1	2	2	2	2	2	1	1	2

Read from policies/stepdown/policy.cpp: 47 observations (base_ang_vel, projected_gravity, velocity_commands, joint_pos_rel, joint_vel_rel, last_action, gait_phase), history 1, 12 actions, owned() = 15.

What reading the adapter found

actions are clipped to +-1 before scaling

Upstream config: g1_config.yaml (unitree_rl_gym).

The policy's contract against the upstream config:
phase clock: contract runs -1 policy steps (-20 ms, -9 deg of a 0.8 s gait) ahead of training (boundary A deviation)
unlisted pose: 6 of 17 joint(s) differ; largest left_elbow_joint contract 0.6 against 0
unlisted kp: 17 of 17 joint(s) differ; largest left_elbow_joint contract 14.25 against 500
unlisted kd: 17 of 17 joint(s) differ; largest left_elbow_joint contract 0.9072 against 10
Port values against the policy's contract: no differences

wcomptonPolicyFalls under its authors' own config (3.4 s of 90): the score belongs to the policy, not the port.2.32.2

Stage	Mean survival (s)	Completed
Authors' config	3.4	0/10
Port's values	3.4	0/10
Harness holds arms	4.4	0/10
Random arm walk	3.9	0/10
Plus punches	2.2	0/10
Port, full benchmark	2.2	0/10
Port, arms still, no punches	4.4	0/10

Run	0	1	2	3	4	5	6	7	8	9
Here	1	2	2	2	4	2	2	2	2	3
Benchmark, MuJoCo	1	2	2	2	4	2	2	2	2	3
Benchmark, PhysX	1	2	1	1	3	2	3	1	2	3

Read from policies/wcompton/policy.cpp: 74 observations (base_ang_vel, projected_gravity, velocity_commands, gait_phase, joint_pos_rel, joint_vel_rel, last_action), history 1, 21 actions, owned() = 13.

Upstream config: policy.onnx metadata (checkpoint).

The policy's contract against the upstream config:
kp: 21 of 21 joints differ; largest relative +4.71% (right_elbow_joint)
  right_elbow_joint: contract 14.2506, the upstream config 13.60974312, diff +0.6409 (+4.71%)
  right_knee_joint: contract 99.0984, the upstream config 103.897522, diff -4.799 (-4.62%)
  right_hip_yaw_joint: contract 40.1792, the upstream config 38.53496933, diff +1.644 (+4.27%)
  left_elbow_joint: contract 14.2506, the upstream config 14.87693882, diff -0.6263 (-4.21%)
  left_ankle_roll_joint: contract 28.5012, the upstream config 29.70834541, diff -1.207 (-4.06%)
  left_hip_yaw_joint: contract 40.1792, the upstream config 41.86675644, diff -1.688 (-4.03%)
  right_hip_pitch_joint: contract 40.1792, the upstream config 41.66600037, diff -1.487 (-3.57%)
  right_ankle_roll_joint: contract 28.5012, the upstream config 29.52899551, diff -1.028 (-3.48%)
  right_shoulder_pitch_joint: contract 14.2506, the upstream config 13.78497601, diff +0.4656 (+3.38%)
  left_shoulder_roll_joint: contract 14.2506, the upstream config 13.86590099, diff +0.3847 (+2.77%)
  waist_yaw_joint: contract 28.5012, the upstream config 29.17500496, diff -0.6738 (-2.31%)
  left_hip_roll_joint: contract 99.0984, the upstream config 97.02146912, diff +2.077 (+2.14%)
  left_ankle_pitch_joint: contract 28.5012, the upstream config 27.9169178, diff +0.5843 (+2.09%)
  left_shoulder_pitch_joint: contract 14.2506, the upstream config 14.53095341, diff -0.2803 (-1.93%)
  right_hip_roll_joint: contract 99.0984, the upstream config 100.3615189, diff -1.263 (-1.26%)
  left_hip_pitch_joint: contract 40.1792, the upstream config 39.68383408, diff +0.4954 (+1.25%)
  left_shoulder_yaw_joint: contract 14.2506, the upstream config 14.42296505, diff -0.1723 (-1.19%)
  right_shoulder_yaw_joint: contract 14.2506, the upstream config 14.39964008, diff -0.149 (-1.03%)
  left_knee_joint: contract 99.0984, the upstream config 98.37251282, diff +0.7259 (+0.74%)
  right_shoulder_roll_joint: contract 14.2506, the upstream config 14.16403484, diff +0.08659 (+0.61%)
  right_ankle_pitch_joint: contract 28.5012, the upstream config 28.59693336, diff -0.09568 (-0.33%)
kd: 21 of 21 joints differ; largest relative +9.60% (right_ankle_pitch_joint)
  right_ankle_pitch_joint: contract 1.81445, the upstream config 1.655584931, diff +0.1589 (+9.60%)
  left_hip_roll_joint: contract 6.3088, the upstream config 5.792602062, diff +0.5162 (+8.91%)
  waist_yaw_joint: contract 1.81445, the upstream config 1.682005405, diff +0.1324 (+7.87%)
  left_shoulder_yaw_joint: contract 0.907223, the upstream config 0.8423577547, diff +0.06487 (+7.70%)
  left_hip_pitch_joint: contract 2.55789, the upstream config 2.770213604, diff -0.2123 (-7.66%)
  left_knee_joint: contract 6.3088, the upstream config 6.813812256, diff -0.505 (-7.41%)
  right_hip_roll_joint: contract 6.3088, the upstream config 6.79916954, diff -0.4904 (-7.21%)
  right_ankle_roll_joint: contract 1.81445, the upstream config 1.947354198, diff -0.1329 (-6.83%)
  left_shoulder_roll_joint: contract 0.907223, the upstream config 0.971396029, diff -0.06417 (-6.61%)
  left_hip_yaw_joint: contract 2.55789, the upstream config 2.413009644, diff +0.1449 (+6.00%)
  left_ankle_pitch_joint: contract 1.81445, the upstream config 1.713458896, diff +0.101 (+5.89%)
  right_hip_yaw_joint: contract 2.55789, the upstream config 2.42055583, diff +0.1373 (+5.67%)
  right_shoulder_yaw_joint: contract 0.907223, the upstream config 0.9585093856, diff -0.05129 (-5.35%)
  left_elbow_joint: contract 0.907223, the upstream config 0.8673850894, diff +0.03984 (+4.59%)
  right_hip_pitch_joint: contract 2.55789, the upstream config 2.681023359, diff -0.1231 (-4.59%)
  right_shoulder_roll_joint: contract 0.907223, the upstream config 0.8824035525, diff +0.02482 (+2.81%)
  right_shoulder_pitch_joint: contract 0.907223, the upstream config 0.9246640205, diff -0.01744 (-1.89%)
  right_elbow_joint: contract 0.907223, the upstream config 0.923692286, diff -0.01647 (-1.78%)
  right_knee_joint: contract 6.3088, the upstream config 6.215536594, diff +0.09327 (+1.50%)
  left_shoulder_pitch_joint: contract 0.907223, the upstream config 0.896725893, diff +0.0105 (+1.17%)
  left_ankle_roll_joint: contract 1.81445, the upstream config 1.807354331, diff +0.007092 (+0.39%)
unlisted pose: same on 8 joint(s)
unlisted kp: same on 8 joint(s)
unlisted kd: same on 8 joint(s)
Port values against the policy's contract: no differences
port: harness drives 8 joint(s) (hold), policy sees them as real: left_shoulder_pitch_joint, right_shoulder_pitch_joint, left_shoulder_roll_joint, right_shoulder_roll_joint, left_shoulder_yaw_joint, right_shoulder_yaw_joint ...

decoupled_wbch066_p000SurvivesStands and tracks in its own setup (90 s) and survives every stage within 5 s of it.84.886.7

Stage	Mean survival (s)	Completed
Port's values	90.0	10/10
Random arm walk	90.0	10/10
Plus punches	86.7	7/10
Port, full benchmark	86.7	7/10
Port, arms still, no punches	90.0	10/10

Run	0	1	2	3	4	5	6	7	8	9
Here	90	90	90	90	90	86	71	90	90	81
Benchmark, MuJoCo	90	90	90	71	90	86	71	90	90	81
Benchmark, PhysX	90	61	90	90	90	90	90	90	90	90

Read from policies/decoupled_wbc/policy.cpp (variant decoupled_wbc_h066_p000): 116 observations (base_ang_vel, projected_gravity, velocity_commands, constant, joint_pos_rel, joint_vel_rel, last_action), history 2, 15 actions, owned() = 15.

What reading the adapter found

4 observation element(s) are constants the port feeds: [67]=0.66, [68]=0, [69]=0, [70]=0

No upstream config read.

Port values against the policy's contract: no differences

decoupled_wbch066_p012SurvivesStands and tracks in its own setup (90 s) and survives every stage within 5 s of it.84.886.7

Stage	Mean survival (s)	Completed
Port's values	90.0	10/10
Random arm walk	90.0	10/10
Plus punches	86.7	7/10
Port, full benchmark	86.7	7/10
Port, arms still, no punches	90.0	10/10

Run	0	1	2	3	4	5	6	7	8	9
Here	90	90	90	90	90	86	71	90	90	81
Benchmark, MuJoCo	90	90	90	71	90	86	71	90	90	81
Benchmark, PhysX	90	90	81	71	90	90	90	90	90	90

Read from policies/decoupled_wbc/policy.cpp (variant decoupled_wbc_h066_p012): 116 observations (base_ang_vel, projected_gravity, velocity_commands, constant, joint_pos_rel, joint_vel_rel, last_action), history 2, 15 actions, owned() = 15.

What reading the adapter found

4 observation element(s) are constants the port feeds: [67]=0.66, [68]=0, [69]=0.12, [70]=0

No upstream config read.

Port values against the policy's contract: no differences

decoupled_wbch070_p000SurvivesStands and tracks in its own setup (90 s) and survives every stage within 5 s of it.82.887.6

Stage	Mean survival (s)	Completed
Port's values	90.0	10/10
Random arm walk	90.0	10/10
Plus punches	87.6	8/10
Port, full benchmark	87.6	8/10
Port, arms still, no punches	90.0	10/10

Run	0	1	2	3	4	5	6	7	8	9
Here	90	90	90	90	90	86	71	90	90	90
Benchmark, MuJoCo	90	61	90	71	90	86	71	90	90	90
Benchmark, PhysX	90	31	90	71	90	90	90	90	90	90

Read from policies/decoupled_wbc/policy.cpp (variant decoupled_wbc_h070_p000): 116 observations (base_ang_vel, projected_gravity, velocity_commands, constant, joint_pos_rel, joint_vel_rel, last_action), history 2, 15 actions, owned() = 15.

What reading the adapter found

4 observation element(s) are constants the port feeds: [67]=0.7, [68]=0, [69]=0, [70]=0

No upstream config read.

Port values against the policy's contract: no differences

gr00t_wbch066_p000SurvivesStands and tracks in its own setup (90 s) and survives every stage within 5 s of it.87.187.1

Stage	Mean survival (s)	Completed
Port's values	90.0	10/10
Random arm walk	90.0	10/10
Plus punches	87.1	9/10
Port, full benchmark	87.1	9/10
Port, arms still, no punches	90.0	10/10

Run	0	1	2	3	4	5	6	7	8	9
Here	90	61	90	90	90	90	90	90	90	90
Benchmark, MuJoCo	90	61	90	90	90	90	90	90	90	90
Benchmark, PhysX	90	90	90	71	90	90	90	90	90	90

Read from policies/gr00t_wbc/policy.cpp (variant gr00t_wbc_h066_p000): 516 observations (velocity_commands, constant, base_ang_vel, projected_gravity, joint_pos_rel, joint_vel_rel, last_action), history 6, 15 actions, owned() = 15.

What reading the adapter found

the port observes motor(s) [15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28] that no action drives (held by the port or the harness): read after the policy's joints
4 observation element(s) are constants the port feeds: [433]=0.66, [434]=0, [435]=0, [436]=0
the port runs model_walk.onnx while the command's norm exceeds 0.05, else model_balance.onnx

No upstream config read.

Port values against the policy's contract: no differences

gr00t_wbch066_p012SurvivesStands and tracks in its own setup (90 s) and survives every stage within 5 s of it.87.187.1

Stage	Mean survival (s)	Completed
Port's values	90.0	10/10
Random arm walk	90.0	10/10
Plus punches	87.1	9/10
Port, full benchmark	87.1	9/10
Port, arms still, no punches	90.0	10/10

Run	0	1	2	3	4	5	6	7	8	9
Here	90	61	90	90	90	90	90	90	90	90
Benchmark, MuJoCo	90	61	90	90	90	90	90	90	90	90
Benchmark, PhysX	90	90	90	71	56	90	90	90	90	90

Read from policies/gr00t_wbc/policy.cpp (variant gr00t_wbc_h066_p012): 516 observations (velocity_commands, constant, base_ang_vel, projected_gravity, joint_pos_rel, joint_vel_rel, last_action), history 6, 15 actions, owned() = 15.

What reading the adapter found

the port observes motor(s) [15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28] that no action drives (held by the port or the harness): read after the policy's joints
4 observation element(s) are constants the port feeds: [433]=0.66, [434]=0, [435]=0.12, [436]=0
the port runs model_walk.onnx while the command's norm exceeds 0.05, else model_balance.onnx

No upstream config read.

Port values against the policy's contract: no differences

Since the last audit

The first audit ran at benchmark commit 6d331a8. There, schoi lost 85 s between its authors' config (90 s) and the port's values (4.7 s): the port contract held its waist at kp about 28, against 300 in the authors' config. The harness now holds every leg and waist motor at the policy's own gains (rhoyn/teleop-walking-benchmark@4ed23c2), and what stops schoi now is the punches.
The same harness fix moves zealot in the benchmark from 8.2 s to 67.6 s. zealot is now read too: it observes the action from two steps back, joint velocities by difference, a gait clock whose period follows the command speed, and a command it steers itself toward the waypoint.
nanog1 (right hip roll clamp) and homie (history start) were corrected upstream. nanog1 and wcompton, unclear in the first audit, now run their authors' configs (nanoG1's deploy_g1.py; the checkpoint's own metadata for wcompton): both fall within 4 s under them, so their scores belong to the policies, not the ports.
Thirteen more ports and variants are read, each rebuilt to 1e-6 or better on random inputs: decoupled_wbc and gr00t_wbc (four variants each; gr00t_wbc switches between its walking and balance networks by the command), homie, falcon, openwbt, asap and handoff; and holosoma's observation now matches its adapter. What they needed: observed joints no action drives, the harness's arm targets, commands a port steers or gates itself (by distance, by a walk latch, by size, after a warm-up), gait clocks that stop or warp, and layouts no single history window describes.
Every stage now runs 10 seeds (3 before), and seed n is the benchmark's run n: the tour, the arm walk and the punches draw from the benchmark's own random streams, so each run here pairs with the same run there. The punches now land as the benchmark's do (at the joint's anchor, in a direction that turns with the link), the arm walk redraws a target that puts a hand behind its stance position or above the head, and every run starts as the benchmark's do (3 s on a crane at the harness's stance, then let go) and ends when the pelvis is below 0.2 m.

Not benched

The reader probes an adapter by running it on the CPU and checks that gaitkeeper rebuilds its observation exactly; it refuses rather than guesses. 9 adapters are refused, and sunny reads but is not benched:

amoTwo networks: an adapter network's output feeds the walking policy, which also keeps a 25-frame history in a second input; its input is not a robot state gaitkeeper can vary. Benchmark: 75.7 s in MuJoCo.bfm_zeroLatent-conditioned: it picks one of 12 latents (loaded at start-up from a file the benchmark's export script writes) by the velocity command, and feeds its network the latent, not the command. Benchmark: 12.3 s in MuJoCo.josabb41 actions, 12 of them hand joints the benchmark's G1 model does not have. Benchmark: 10.1 s in MuJoCo.mimic_liteA motion-tracking stack (a planner network feeding an actor), not a velocity-command walker. Benchmark: 48.1 s in MuJoCo.mturan33Observes roll, pitch and yaw computed from the base quaternion with its components taken in (x, y, z, w) order from a (w, x, y, z) quaternion, as its own training script does (so its roll reads pi when upright); the port matches its training, but gaitkeeper has no term for that observation yet. Benchmark: 2.8 s in MuJoCo.run_residualTwo networks (a motion generator and a residual actor). Benchmark: 51.7 s in MuJoCo.sonicA motion-tracking stack (a planner network and an encoder-decoder pair), not a velocity-command walker. Benchmark: 68.5 s in MuJoCo.sunnyReads, and its observation matches the adapter's; its weights are not published, so it is not benched. Benchmark: 7.3 s in MuJoCo.wbc_agileBuilds its network through TensorRT's own API from the ONNX weights instead of the benchmark's engine interface, so it does not build for the CPU. Benchmark: 70.9 s in MuJoCo.wbc_agile_velocityFeeds its graph twelve inputs (raw joint states, a quaternion built from gravity) and keeps its observation history inside the graph, fed back from its outputs; gaitkeeper feeds one observation input. Benchmark: 75.1 s in MuJoCo.

How to read a verdict

Every runSeed n draws the benchmark’s run n: its waypoints, arm targets and punches, from its own random streams. A run starts as the benchmark’s do, 3 s on a crane at the harness’s stance and then let go, and ends when the pelvis is below 0.2 m. Authors' configThe policy as its authors trained or deployed it: their gains, poses and observation, read from their config (Isaac Lab env.yaml, unitree_rl_gym, a Unitree deploy.yaml). Run with the same drive as the port, so stages differ only in values. Port's valuesThe values the benchmark's adapter feeds and applies, with the policy driving every joint it lists. Harness holds armsThe harness takes the arm joints from the policy and holds them at its stance with its own gains. Random arm walkThe harness moves the arms at random, as teleoperation does. Plus punchesA punch on a random link every waypoint, rising to 500 N. Port, full benchmarkAll of the above with the port's own view of the arms (some ports hide arm motion from the policy). Bench, HereMean survival over runs 0–9: the benchmark’s MuJoCo numbers for those run ids, and the port under the full benchmark here on the same draws. A row’s run table marks a run here or in the benchmark’s PhysX that ends more than 1 s from the benchmark’s MuJoCo run.

Evidence L1: one runner, one model, stated assumptions. A verdict names the step that costs survival here; it is not a claim about the benchmark's own numbers, and a step that is not run cannot be blamed.

Reproduce

pip install "gaitkeeper[sim]==0.5.2"
git clone https://github.com/rhoyn/teleop-walking-benchmark twb   # at 106faa0

git clone https://github.com/unitreerobotics/unitree_rl_gym   # rl_gym's authors' config

# read a port from its adapter (needs g++ or clang++); a port with variants reads each
gaitkeeper adapter twb/policies/rl_gym/policy.cpp --mjcf twb/assets/g1_29dof.xml --out contracts/
gaitkeeper adapter twb/policies/gr00t_wbc/policy.cpp --mjcf twb/assets/g1_29dof.xml --out contracts/

# run the ladder, with the authors' config as the first stage when there is one
gaitkeeper bench --adapter twb/policies/rl_gym/policy.cpp --onnx twb/policies/rl_gym/model.onnx \
    --upstream unitree_rl_gym/deploy/deploy_mujoco/configs/g1.yaml \
    --mjcf twb/assets/g1_29dof.xml --seeds 10 --md rl_gym.md

# a variant, by name (its models are found next to the one given)
gaitkeeper bench --adapter twb/policies/gr00t_wbc/policy.cpp --variant gr00t_wbc_h066_p012 \
    --onnx twb/policies/gr00t_wbc/model_walk.onnx --mjcf twb/assets/g1_29dof.xml --seeds 10

Source, method and every command: github.com/dundysm/gaitkeeper.

Interactive: command dead zones of the unitree_rl_lab G1 policy, drag the command and watch it stand or walk.

Run it on your own policy

The same checks on a G1 policy whose config gaitkeeper reads, in one to two and a half minutes on a laptop:

gaitkeeper doctor --onnx policy.onnx --config <your config> --mjcf scene.xml

The config is read by content: an Isaac Lab env.yaml, a Unitree deploy.yaml, a unitree_rl_gym or legged_gym config, or a contract. Doctor reports the contract fields no file states, the commands the policy ignores, and survival on the tour with the arms its own, moved at random and punched.

Help

A verdict that looks wrong, a config gaitkeeper cannot read, or harness logs for a blind test are all useful. Good first issues · Contributing · Discussions