Treating the Symptom, Missing the Cause: Anatomy of a Scientific Reasoning Failure
cs.AI
Submitted: 2026-11-03
Updated: 2026-11-05
Comments: 18 pages (8 pages main text, remainder appendix), 9 figures, 6 tables. NeurIPS 2026 Workshop on Failure Modes of AI in Biology (ICBINB-BIO); non-archival
Code: https://github.com/RelationRx/luca
License: http://creativecommons.org/licenses/by/4.0/
The gist: Scientific agents can find interventions that work without understanding why they do.
Terminology
Abstract
Scientific agents can find interventions that work without understanding why they do. We study this failure mode in controlled synthetic regulatory systems, where agents must experimentally identify a hidden fault and rescue the resulting phenotype. Across seven LLMs, every model restores the healthy phenotypes more often than it identifies the fault. Claude Opus 4.6 and Sonnet 4.6 rescue every organism while diagnosing only 55% and 35% of them. This failure is not simply due to insufficient exploration or unavailable evidence. In most cases, the cause is experimentally identifiable, and agents encounter evidence that distinguishes competing causes in 91% of relevant sessions, and still frequently misdiagnose the system. Targeted probes reveal where experimental reasoning breaks down. Models reliably interpret decisive evidence and revise hypotheses when the causal comparison is explicit, and can locate that evidence within a full experimental record. However, they struggle to predict the outcome of the experiment under each competing cause, a necessary step in using the observed result to distinguish between them. Agents can therefore gather the right evidence and still infer the wrong cause. As AI systems take on a growing role in synthesising scientific evidence, these findings highlight the need to evaluate how they use evidence to reach causal conclusions, alongside whether their interventions succeed.
Sources
- The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery
- DiscoveryBench: Towards Data-Driven Discovery with Large Language Models
- Tiny Moves: Game-based Hypothesis Refinement
- HypoSpace: A Diagnostic Benchmark for Set-Valued Hypothesis Generation under Underdetermination and Sublinear Coverage Bounds
- LAB-Bench: Measuring Capabilities of Language Models for Biology Research
- Contextualizing biological perturbation experiments through language
- BioMaze: Benchmarking and Enhancing Large Language Models for Biological Pathway Reasoning
- ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery
- AI scientists produce results without reasoning scientifically
- MedEinst: Benchmarking the Einstellung Effect in Medical LLMs through Counterfactual Differential Diagnosis
- CausalGame: Benchmarking Causal Thinking of LLM Agents in Games
- CausaLab: A Scalable Environment for Interactive Causal Discovery Toward AI Scientists
- BoxingGym: Benchmarking Progress in Automated Experimental Design and Model Discovery
- LUCA: A Synthetic World for Experimental Reasoning
- What Do LLM Agents Know About Their World? Task2Quiz: A Paradigm for Studying Environment Understanding
- Failing to Falsify: Evaluating and Mitigating Confirmation Bias in Language Models
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents