When the Simulator Fails,
Not the Policy
Many failures in simulated robot benchmarks are the simulator's fault, not the policy's. We find the cause and fix it.
The work in under two minutes.
The policy places the can.
The simulator throws it out.
LingBot-VA puts a 10 g can into the basket. No robot link touches it, and the fastest link moves at under 0.1 m/s. Then, within one 4 ms physics step, the can speeds up from 0.2 to 22 m/s. The benchmark records a failure.
An auditing plugin.
The evaluation stays the same.
SimuGuard attaches to the benchmark's own evaluation script on LIBERO, RoboTwin, RoboCasa and RoboDojo. It records every physics step, flags objects that move faster than the robot or gravity can explain, replays any episode bit for bit, and reruns it with one change.
Every step
State, controls and contacts at every physics step, with no change to the evaluation.
Three stages
A contact that starts motion, then a gravity check, then a carrier check. 37 of 40 flagged clips judged implausible by five engineers.
Bit for bit
Every recorded episode reproduces exactly, so a failure can be studied step by step.
One change
Same seed, same actions, one simulation parameter changed, scored by the benchmark's own success check.
One constant.
The two convex pieces that start the ejection are 21 mm apart. PhysX 5.3.1 reports them 27.5 mm deep, with the normal pointing the wrong way, and the solver removes that depth in one step.
Its GJK distance algorithm discards a triangle of its simplex when the area test below is true. The threshold is fixed in meters. RoboTwin's containers are cut into pieces one to two centimeters across, so valid triangles fall under it.
| Engine | False contacts | Real | Expert artifacts |
|---|---|---|---|
| PhysX 5.3.1 (RoboTwin) | 20 / 20 | 4 / 4 | 18 · 6 |
| 5.3.1 + threshold of 5.4.0 | 0 / 20 | 4 / 4 | 1 · 0 |
| PhysX 5.9.0 | 0 / 20 | 4 / 4 | — |
Change that constant.
Nothing else.
LingBot-VA successes of 100 closed loop episodes on five container tasks. Episodes with an artifact fall from 12 to 0. Each particle cell is one episode, read from the run manifests: benchmark engine on the left, repaired engine on the right.
Successes of 20, artifacts in parentheses. Offset: contact offset 1 mm, engine unchanged. Repaired: threshold of PhysX 5.4.0.
Thin walls, small pieces.
Failures with an artifact gather in five tasks that place an object into a container with thin walls. Each dot is one failed LingBot-VA episode on RoboTwin 2.0.
Place Can Basket 21/23 · Stack Bowls Three 12/12
on eight of them
Not only RoboTwin.
On every benchmark we tested, an object at rest can be thrown in a single physics step, though nothing pushes it that hard. On RoboTwin we traced it to the GJK threshold. RoboDojo runs Isaac Sim with contact generation on the GPU and also shows a false contact, on a path we have not located. On MuJoCo, these events are few and slow.
RoboTwin 2.0
The can rests in the basket. In one 4 ms step it is thrown out.
RoboDojo
Insert Tubes: a tube is thrown at 4.31 m/s.
LIBERO
Mug and microwave are 1.4 mm apart. MuJoCo reports them 242.8 mm deep, and the mug is thrown at 1.57 m/s.
| Benchmark | Engine | Episodes | Artifact | Peak m/s |
|---|---|---|---|---|
| RoboTwin 2.0 | PhysX · SAPIEN | 2060 | 80 | 21.8 |
| RoboDojo | PhysX · Isaac Sim | 40 | 7 | 6.30 |
| LIBERO | MuJoCo | 400 | 2 | 1.57 |
| RoboCasa | MuJoCo | 300 | 5 | 1.19 |
A success rate in simulation counts the simulator's failures too.
Benchmarks on PhysX 5.3.1 or earlier should rerun in a repaired engine. Where the engine cannot change, report the success rate together with the range it spans when every flagged episode fails or passes.
LingBot-VA, five container tasks: reported 83.1%, replay estimate 91.9%, excluding flagged episodes 98.0%.
BibTeX
@article{zhou2026simuguard,
title = {When the Simulator Fails, Not the Policy:
Auditing Contact Artifacts in Simulated Robot Benchmarks},
author = {Zhou, Alex and Wu, Junhao and Zhou, Xueyang and Huo, Yixiao and
Gao, Hongcheng and Zhu, Di and Zhang, Zhenghao and Chen, Yongchao},
journal = {arXiv preprint},
year = {2026}
}