When the Simulator Fails, Not the Policy: Auditing Contact Artifacts in Simulated Robot Benchmarks
cs.RO
Submitted: 2026-10-30
Updated: 2026-10-30
Comments: https://github.com/ALEXander22568/simuguard
Code: https://github.com/ALEXander22568/simuguard
Project page: https://nvidia-omniverse.github.io/PhysX/physx/5.4.1/docs/BestPractices.html
License: http://creativecommons.org/licenses/by/4.0/
The gist: Simulation benchmarks score robot policies by success rate, and every failed episode counts against the policy.
Terminology
Abstract
Simulation benchmarks score robot policies by success rate, and every failed episode counts against the policy. Many of these failures are the simulator's fault, not the policy's; we find the cause and fix it. Testing a vision language action model and a world action model in simulation, we find that the policy often places the object and the simulator then ejects it, and that the benchmark's own scripted demonstrations fail the same way. We present SimuGuard, an auditing plugin for LIBERO, RoboTwin, RoboCasa and RoboDojo. It leaves the evaluation unchanged. At every physics step, it records the state and flags an object that starts moving at a contact faster than the robot's contact or gravity account for. It replays a recorded episode bit for bit, and it reruns the episode with one simulation parameter changed. With it we trace these episodes on RoboTwin to the collision detection of the physics engine: for geometry one to two centimeters apart, the engine reports a penetration of about two centimeters, and the solver throws the object out at several meters per second. The cause is one constant: the distance algorithm discards triangles below a fixed area threshold, which valid triangles fall below at the centimeter scale. With that constant changed and nothing else, episodes with an artifact fall from 12 in 100 to none, and the success rate on five container tasks rises from 89% to 98%. Success rates in simulation thus count failures that the simulator causes. Benchmarks on PhysX 5.3.1 or earlier should rerun in a repaired engine. Where the engine cannot change, they should report the success rate together with the range it spans when every flagged episode fails or passes: 75.8% to 98.5% around a reported 83.1% for LingBot-VA.
Sources
- RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies
- GR00T N1: An Open Foundation Model for Generalist Humanoid Robots
- $\pi_0$: A Vision-Language-Action Flow Model for General Robot Control
- RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
- GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulation
- RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulation
- RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies
- SafeVLA-Bench: A Benchmark for the Success-Safety Gap in Vision-Language-Action Models
- XR-1: Towards Versatile Vision-Language-Action Models via Learning Unified Vision-Motion Representations
- LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Models
- Towards Generalizable Vision-Language Robotic Manipulation: A Benchmark and LLM-guided 3D Policy
- Gemini Robotics: Bringing AI into the Physical World
- ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills
- What Are We Actually Benchmarking in Robot Manipulation?
- OpenVLA: An Open-Source Vision-Language-Action Model
- Robot Learning as an Empirical Science: Best Practices for Policy Evaluation
- Contact Models in Robotics: a Comparative Analysis
- Causal World Modeling for Robot Control
- Evaluating Real-World Robot Manipulation Policies in Simulation
- LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning
Related papers
- Rethinking Actions: Continuity-Aware Priors for World-Model Planning
- MOSAIC: Modular Obstacle and Spatial-Aware Framework for Manipulation In Cluttered Scenes
- SAGAS: Semantic-Aware Graph-Assisted Stitching for Offline Temporal Logic Planning
- DexSeed: Expanding Human Hand-Object Interactions into Robot Demonstrations for Dexterous Policy Learning
- LaST0.5: A Generalist World Action Model with Interaction-Centric Latent Reasoning
- CIALLO: Learning Shared Intent Representations for Cross-Embodiment Manipulation