SimuGuard GitHub ↗
Replay · bit exactREC
taskPlace Can Basket
policy · seedLingBot-VA · 100019
physics step—
sim time—
can speed—
can–basket contact pairs—
deepest reported contact—
Preprint · 2026

When the Simulator Fails,
Not the Policy

Many failures in simulated robot benchmarks are the simulator's fault, not the policy's. We find the cause and fix it.

Alex Zhou1 · Junhao Wu1 · Xueyang Zhou1 · Yixiao Huo1 · Hongcheng Gao1 · Di Zhu2 · Zhenghao Zhang2 · Yongchao Chen1,†
1College of AI, Tsinghua University   2University of Chinese Academy of Sciences   †Corresponding author
Scroll — the particles are the real collision geometry
Video

The work in under two minutes.

Video coming soon
01 · The problem

The policy places the can.
The simulator throws it out.

LingBot-VA puts a 10 g can into the basket. No robot link touches it, and the fastest link moves at under 0.1 m/s. Then, within one 4 ms physics step, the can speeds up from 0.2 to 22 m/s. The benchmark records a failure.

can speed, m/s · recorded episode
one dot = one 4 ms physics step
40 / 44
LingBot-VA failures on five container tasks that contain a contact artifact
6 / 10
HyVLA failures on the same tasks that contain one
2.4 J
kinetic energy the can gains in one step, from no source
16 / 22
seeds RoboTwin skipped because its own scripted demonstration failed, that contain one
02 · SimuGuard

An auditing plugin.
The evaluation stays the same.

SimuGuard attaches to the benchmark's own evaluation script on LIBERO, RoboTwin, RoboCasa and RoboDojo. It records every physics step, flags objects that move faster than the robot or gravity can explain, replays any episode bit for bit, and reruns it with one change.

01 · RECORD

Every step

State, controls and contacts at every physics step, with no change to the evaluation.

02 · DETECT

Three stages

A contact that starts motion, then a gravity check, then a carrier check. 37 of 40 flagged clips judged implausible by five engineers.

03 · REPLAY

Bit for bit

Every recorded episode reproduces exactly, so a failure can be studied step by step.

04 · RERUN

One change

Same seed, same actions, one simulation parameter changed, scored by the benchmark's own success check.

03 · The cause

One constant.

The two convex pieces that start the ejection are 21 mm apart. PhysX 5.3.1 reports them 27.5 mm deep, with the normal pointing the wrong way, and the solver removes that depth in one step.

Its GJK distance algorithm discards a triangle of its simplex when the area test below is true. The threshold is fixed in meters. RoboTwin's containers are cut into pieces one to two centimeters across, so valid triangles fall under it.

PhysX 5.3.1|ab × ac|² ≤ 2−23 ≈ 1.19×10−7
at the onsets6.0×10−8 … 1.189×10−7
PhysX 5.4.0|ab × ac|² ≤ 2−46
EngineFalse contactsRealExpert artifacts
PhysX 5.3.1 (RoboTwin)20 / 204 / 418 · 6
5.3.1 + threshold of 5.4.00 / 204 / 41 · 0
PhysX 5.9.00 / 204 / 4—
04 · The fix

Change that constant.
Nothing else.

89→98

LingBot-VA successes of 100 closed loop episodes on five container tasks. Episodes with an artifact fall from 12 to 0. Each particle cell is one episode, read from the run manifests: benchmark engine on the left, repaired engine on the right.

successfailurefailure with artifactsuccess with artifact

Successes of 20, artifacts in parentheses. Offset: contact offset 1 mm, engine unchanged. Repaired: threshold of PhysX 5.4.0.

05 · Where it happens

Thin walls, small pieces.

Failures with an artifact gather in five tasks that place an object into a container with thin walls. Each dot is one failed LingBot-VA episode on RoboTwin 2.0.

40 / 44
five container tasks
Place Can Basket 21/23 · Stack Bowls Three 12/12
12 / 146
the other 45 tasks
on eight of them
06 · Four benchmarks, two engines

Not only RoboTwin.

On every benchmark we tested, an object at rest can be thrown in a single physics step, though nothing pushes it that hard. On RoboTwin we traced it to the GJK threshold. RoboDojo runs Isaac Sim with contact generation on the GPU and also shows a false contact, on a path we have not located. On MuJoCo, these events are few and slow.

Clip coming soon

RoboTwin 2.0

SAPIEN · PhysX 5.3.1 · LingBot-VA

The can rests in the basket. In one 4 ms step it is thrown out.

80 of 2060 episodes · peak 21.8 m/s
Clip coming soon

RoboDojo

Isaac Sim · GPU PhysX · XR-1

Insert Tubes: a tube is thrown at 4.31 m/s.

7 of 40 episodes · peak 6.30 m/s
Clip coming soon

LIBERO

MuJoCo · π0.5

Mug and microwave are 1.4 mm apart. MuJoCo reports them 242.8 mm deep, and the mug is thrown at 1.57 m/s.

2 of 400 episodes · peak 1.57 m/s
Each clip is replayed from the recorded episode, 2× slower than real time.
BenchmarkEngineEpisodesArtifactPeak m/s
RoboTwin 2.0PhysX · SAPIEN20608021.8
RoboDojoPhysX · Isaac Sim4076.30
LIBEROMuJoCo40021.57
RoboCasaMuJoCo30051.19
07 · What to report

A success rate in simulation counts the simulator's failures too.

Benchmarks on PhysX 5.3.1 or earlier should rerun in a repaired engine. Where the engine cannot change, report the success rate together with the range it spans when every flagged episode fails or passes.

LingBot-VA, five container tasks: reported 83.1%, replay estimate 91.9%, excluding flagged episodes 98.0%.

Cite

BibTeX

@article{zhou2026simuguard,
  title   = {When the Simulator Fails, Not the Policy:
             Auditing Contact Artifacts in Simulated Robot Benchmarks},
  author  = {Zhou, Alex and Wu, Junhao and Zhou, Xueyang and Huo, Yixiao and
             Gao, Hongcheng and Zhu, Di and Zhang, Zhenghao and Chen, Yongchao},
  journal = {arXiv preprint},
  year    = {2026}
}