Adaptive Experimentation Can Hide Model Error:Detection Limits and Falsification Budget
stat.ME, cs.AI
Submitted: 2026-11-05
Updated: 2026-11-05
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
The gist: An experiment can be informative about the parameters of a working model and uninformative about whether that model is wrong.
Terminology
Abstract
An experiment can be informative about the parameters of a working model and uninformative about whether that model is wrong. Adaptive design can reinforce this mismatch by repeatedly selecting settings where an incorrect model fits. We bound the power of any valid adequacy test in terms of the experimenter's own sampling distribution, then distinguish information absent from the selected data from information lost by the detector. A computable diagnostic measures unresolved error relative to a declared expansion of the working model. It motivates a training-free rule that reserves part of the design budget for falsification. The empirical record is mixed: the independently specified primary study fails detection eligibility, and a later numerator diagnostic fails its improvement criteria. After the frozen rule fails on fresh continuous-null panels, a disclosed secondary horizon-calibrated revision detects 35.8% of misspecified campaigns versus 31.7% for a class-separation control at T = 240, but has no resolved advantage at T = 480. In a corrected 289-unit measured replay it improves on random design and trails class-discrimination EIG. A conditional futility bound explains why remaining time matters without proving the gate constant or its power. These results support separating parameter learning, model checking and detector performance; they do not establish a universally superior design policy or prospective scientific discovery.
Sources
- Bayesian Model Discovery with Language Models: Closing the Loop between Hypotheses and Experiments
- Strong duality for the GROW criterion
- Learning to Bet for Horizon-Aware Anytime-Valid Testing
- Maximin Robust Bayesian Experimental Design
- On Stopping Times of Power-one Sequential Tests: Tight Lower and Upper Bounds
- Adaptive clinical trials based on design-optimal e-values with automatic curtailment: An application to single-arm trials with binary data
- Robust Experimental Design via Generalised Bayesian Inference
- Metrics for Bayesian Optimal Experiment Design under Model Misspecification
- A Short Note on Almost Sure Convergence of Bayes Factors in the General Set-Up
- Sequential Controlled Sensing for Composite Multihypothesis Testing
- Type-II Error Bounds for Test Supermartingales from Lower-Tail Hypotheses
- Deep Adaptive Design: Amortizing Sequential Bayesian Experimental Design
- BoxingGym: Benchmarking Progress in Automated Experimental Design and Model Discovery
- Automated Hypothesis Validation with Agentic Sequential Falsifications
- Failing to Falsify: Evaluating and Mitigating Confirmation Bias in Language Models
- Confidence Estimation via Sequential Likelihood Mixing
- Learning to Outgrow a Theory: Experimental Discovery Beyond the Initial Hypothesis Space
- Modern Bayesian Experimental Design
- LLM-IDEA: Identifiability-Driven Experimental Agent for Autonomous Discovery of Mechanistic World Models
Related papers
- Learning Robust Treatment Rules for Censored Data
- Neural Estimation of Summary Statistics for Simulation-Based Inference with Non-Asymptotic Guarantees
- Spectral divide-and-conquer MCMC for long stationary time series
- Improving the Efficiency of Subgroup Analysis in Randomized Controlled Trials with TMLE
- Personalized Federated Vector Autoregression with Personalization Diversity
- Doubly Robust and Efficient Calibration of Prediction Sets for Right-Censored Time-to-Event Outcomes