Stand-In Effects and Pooling Amplification: Lessons from Biological World Models for Testing Action Effects
cs.LG, q-bio.QM, stat.ME
Submitted: 2026-11-01
Updated: 2026-11-11
Comments: 119 pages. v4: adds a registered test on recorded robot video (OSF 4wj7k), an over-betting analysis with one new proposition, and a site-count-adaptive check on the ERK fields; corrects a sentence in the abstract
License: http://creativecommons.org/licenses/by/4.0/
The gist: Whether an action-conditioned world model responds to actions is usually checked by replacing or removing the actions and measuring the loss in accuracy.
Terminology
Abstract
Whether an action-conditioned world model responds to actions is usually checked by replacing or removing the actions and measuring the loss in accuracy. Taking biological experiments as the guiding case, we show that such diagnostics report action effects where there are none: under autocorrelated designs, future actions stand in for history the model does not see. When the design law is known, a conditional randomization test written as an e-value is valid in finite samples whatever the model. A second failure arises whenever many sites share one action sequence, as in world models: the pooled score turns small model errors along the action contrast into an e-value near zero, so a valid test loses its power although the effect is real. Tempering the score changes the bet, not the validity. The log e-value is concave in the tempering factor and zero at zero, so too small a factor costs little and too large a one can cost everything; every single factor we tried, estimated or tuned, failed in some setting, and mixtures over a fixed grid never did. Our own preregistered analysis of optogenetic ERK experiments failed by pooling; a tempered statistic, registered before the data were obtained, then rejected the null on a never-opened field (E = 1.1e10). In three registered tests on models trained by others, the pooled test rejected in 0 of 96 units of a diffusion model, and reached the power bar in 10 of 26 Atari games under IRIS's token likelihood and 1 of 6 control tasks under TD-MPC2; mixtures fixed in advance reached it in 96 units, 24 games and 6 tasks. On recorded robot video with iVideoGPT the mixture rejected in 155 of 159 trajectories; the pooled test did not collapse there (143). Under exact nulls the statistics made 14 rejections in 26,664 tests.
Sources
- Trustworthy Feature Importance Avoids Unrestricted Permutations
- Do Robotic World Models Really Follow Actions? Diagnosing and Aligning Action-Conditioned Generation for Policy Learning
- WorldSimProbe: Diagnosing Simulator Faithfulness in Action-Conditioned World Models for Embodied Manipulation
- ML-assisted Randomization Tests for Detecting Treatment Effects in A/B Experiments
- Dueling World Models: Advantage-Style Action Channels for Common-Mode Distractor Rejection
- Randomization Tests in Switchback Experiments
- Validate the Dream Before You Trust Its Verdict: Admissibility for World-Model Simulators
- MiraBench: Evaluating Action-Conditioned Reliability in Robotic World Models
- WorldExam: Benchmarking World Models from Apparent Appearance to Inherent Reactivity
- Design-Ignoring versus Design-Respecting World Models for Epidemiology
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis