WorldSimProbe: Diagnosing Simulator Faithfulness in Action-Conditioned World Models for Embodied Manipulation

The paper introduces WorldSimProbe, a diagnostic benchmark for evaluating action-conditioned world models (ACWMs) in robotic manipulation. The authors argue that "Action-conditioned world models (ACWMs) promise to provide embodied AI with scalable predictive simulators for planning, policy evaluation, and data generation. Realizing this promise requires precise action-conditioned transitions rather than merely plausible outputs. However, prevailing evaluations emphasize visual quality, task outcomes, or coarse rollout-level responsiveness without directly testing simulator fidelity."

The paper formalizes the Observable Simulator Contract, described as a minimal contract that any action-conditioned physical simulator should satisfy: supplied actions must induce corresponding agent motion, and environment responses must be grounded in that realized motion. This defines a simulator chain from action realization to interaction dynamics, including realized agent motion and environment response.

The contract requires two linked consistency conditions:

1. Action-realization consistency: the generated agent motion to correspond to the supplied action under the current scene

2. Interaction-response consistency: the generated environment response to be physically supported by the realized agent motion and initial environment state

The paper emphasizes that "A rollout satisfies the simulator contract only when both links hold jointly: the supplied action is realized as corresponding agent motion, and environment changes are mediated by that motion and its contact events. This distinguishes simulator faithfulness from visual plausibility or task consistency alone."

WorldSimProbe operationalizes the contract through five controlled suites:

This suite isolates this local regime by holding the task and outcome fixed while applying fine-grained action perturbations that induce subtle but measurable trajectory changes. Starting from successful action trajectories, the benchmark constructs triplets with original, small-perturbation, and large-perturbation variants. The evaluation uses oracle-relative score comparing model response ratios to simulator reference ratios, which reduces sensitivity to model-specific visual-error scale and evaluates local response calibration.

This suite evaluates action realization beyond task-associated trajectories through cross-task receiver–donor replay. The evaluation uses Robot-Masked Flow Alignment (RMFA), which compares generated and reference flow vectors at corresponding reference-arm pixels and is sensitive to errors in robot-motion direction, magnitude, and spatial alignment.

This suite tests whether action realization preserves source-specific behavior using "successful and unsuccessful task-directed trajectories from multiple human teleoperators and early- and late-training policy checkpoints, spanning variation in speed, smoothness, corrective behavior, and execution strategy."

The paper notes that "interaction-grounding failures were instead dominated by contact hallucination, where a model generates agent motion or an environment response as though contact occurred despite being unsupported by the supplied action and scene. Three no-contact settings are evaluated: spatial proximity places an object near the trajectory; appearance-induced false contact preserves contact-like visual evidence but removes contact-enabling control; and distractor cases add plausible objects that should remain stationary."

This suite evaluates representative interaction primitives that capture characteristic relationships between agent motion and environment response. The eight primitives are: push, pull, drag, rotate, shake, tap, knock-over, and drop. A VLM judge (Qwen3-VL-8B) predicts the realized primitive, with validation showing the VLM judge agrees with the majority human label on 94–97% of instances across simulators.

The benchmark evaluates six open-source ACWMs across three platforms:

• RoboTwin (50 bimanual manipulation tasks, Aloha-AgileX embodiment)

• ManiSkill (11 single-arm manipulation tasks, Panda embodiment)

• LIBERO (40 single-arm manipulation tasks, Panda embodiment)

The models include action-injection architectures (IRASim, Ctrl-World, BWM, DreamDojo) and unified action–video architectures (LingBot-VA, Cosmos-3-Nano). The final benchmark comprises approximately 5,500 RoboTwin, 6,600 ManiSkill, and 6,500 LIBERO evaluation instances, totaling more than 18,000.

The results show substantial cross-platform ranking consistency (mean pairwise Spearman ρ = 0.695). LingBot-VA leads RoboTwin and ManiSkill, whereas Ctrl-World leads LIBERO and ranks second elsewhere. Importantly, architecture alone does not explain simulator faithfulness, and strength at one simulator-chain stage does not imply overall fidelity.

• Local calibration: "the simulator oracle responds progressively to increasing perturbations, whereas model responses exhibit varying degrees of attenuation, indicating that detecting action changes does not guarantee calibrated response scaling."

• Global coverage: action-realization fidelity declines overall for all six models as receiver–donor motion mismatch increases, with an average Spearman correlation of ρ = −0.433, consistent with models relying increasingly on scene- or task-associated motion priors when supplied controls depart from familiar trajectories.

• Source preservation: All six models score 10.8–16.1 points higher on late- than early-policy checkpoints, while human trajectories exhibit consistent operator-dependent variation.

• Grounding: grounding is strongest for distractors (75.0) and proximity (54.5) but drops for false contact (38.6). Models resist distractors and proximity better than visual contact cues lacking control support.

• Dynamics: Fidelity varies more across primitives than models. Tap is strongest (40.8–56.9), followed by drag, drop, and push; pull, rotate, and knock-over are weaker, while shake is nearly absent (0.0–1.2).

The paper validates WorldSimProbe through human agreement studies: "The VLM predicts positive action following for 91.7% of Diverse/OOD samples versus 42.2% by humans, yielding 0.450 agreement. Under these controls, RMFA remains strongly aligned (ρ = 0.750), whereas IDM falls to ρ = 0.284."

A downstream synthetic-data study showed that "Policies trained on the corresponding synthetic data have standard trajectories clustered at 78–86% success, whereas OOD trajectories separate Ctrl-World, BWM, and Cosmos-3-Nano at 53%, 34%, and 21%, respectively, consistent with the WorldSimProbe ordering."

The paper concludes that "five suites expose weaknesses overlooked by coarse, task-associated protocols. Across six open-source ACWMs and three platforms, the evaluation reveals that apparent action responsiveness can mask miscalibrated realization, fidelity degrades systematically under trajectory and behavior-source shifts, and interaction failures exhibit recurring structure across contact cues and dynamics primitives. The framework provides a transparent, and standardized paradigm for diagnosing ACWM simulator fidelity beyond coarse, task-directed evaluation."

Improvements to AI systems:

1. Action-calibrated world models: Train ACWMs with a loss term that penalizes the ratio between action-perturbation magnitude and resulting trajectory displacement, using oracle-relative scoring (as in Task 1) to enforce proportional response scaling rather than binary action detection.

2. Motion-prior regularization: Add a training objective that minimizes the Spearman correlation drop (currently ρ = −0.433) between action-input deviation and flow-alignment error, forcing the model to rely on supplied actions rather than scene/task priors when controls are OOD (Task 2).

3. Behavior-source disentanglement: Incorporate a domain-adversarial or contrastive loss that separates action-realization from teleoperator/policy-checkpoint style, preventing the model from collapsing source-specific motion patterns (Task 3).

4. Contact-hallucination suppression: Introduce a no-contact grounding loss that penalizes predicted environment responses or agent motion when the action and scene geometry do not support physical contact, targeting the false-contact failure mode (Task 4).

5. Primitive-aware dynamics heads: Add a lightweight classifier or auxiliary task that predicts the interaction primitive (push, pull, rotate, shake, etc.) from latent states, with a training loss that enforces distinct dynamics for each primitive—especially improving weak primitives like shake (currently 0.0–1.2) and knock-over.

6. VLM-judge distillation: Use the validated VLM judge (94–97% human agreement) as a reward model in RL fine-tuning of ACWMs, providing dense feedback on primitive realization and interaction grounding during rollout generation.

What the improved AI system can do:

• Generate rollouts where action magnitude maps linearly to robot motion, even for small perturbations, enabling reliable fine-grained planning.

• Maintain action-following fidelity when given novel or cross-task trajectories, reducing reliance on memorized task priors.

• Preserve operator-specific or policy-specific motion styles (speed, smoothness, corrections) while still executing the commanded action.

• Avoid phantom contacts—no spurious object movement or agent motion when contact is not physically supported.

• Produce realistic dynamics for all eight interaction primitives, including rare ones like shaking and knocking over, with distinct and correct physical responses.

• Self-evaluate its own rollouts for simulator faithfulness using the VLM judge, enabling closed-loop correction during data generation for downstream policy training.

Action-conditioned world models (ACWMs) promise to provide embodied AI with scalable predictive simulators for planning, policy evaluation, and data generation. Realizing this promise requires precise action-conditioned transitions rather than merely plausible outputs. Yet their applicability remains difficult to establish because prevailing evaluations emphasize visual quality, task outcomes, or coarse rollout-level responsiveness without directly testing simulator fidelity. To address this gap, we evaluate ACWMs through the observable capabilities expected of physical simulators. Accordingly, we formalize Observable Simulator Contract, a minimal contract that any action-conditioned physical simulator should satisfy: supplied actions must induce corresponding agent motion, and environment responses must be grounded in that realized motion. To operationalize this contract, we introduce WorldSimProbe, comprising five controlled suites spanning local control sensitivity, global trajectory variation, source-diverse actions, interaction grounding, and dynamics. Suite-specific evaluators assess simulator-relative calibration, dense action-to-motion correspondence, false-interaction grounding, and primitive-level dynamics. We evaluate six open-source ACWMs on more than 18,000 instances across RoboTwin, ManiSkill, and LIBERO. World-SimProbe reveals systematic action-realization degradation across control variation, structured failures in interaction grounding and dynamics, and benchmark signals consistent with human judgments and downstream outcomes. Together, this capability-based framework provides a transparent, and standardized paradigm for diagnosing ACWM simulator fidelity beyond coarse, task-directed evaluation.