Reasoning Models Are Accurate but Unsound on Identification
cs.AI, cs.CR
Submitted: 2026-10-02
Updated: 2026-10-02
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
The gist: A reasoning model asked whether a causal effect is recoverable from observational data can fail in two ways: it refuses an identifiable query or answers a nonidentifiable one.
Abstract
A reasoning model asked whether a causal effect is recoverable from observational data can fail in two ways: it refuses an identifiable query or answers a nonidentifiable one. The latter is more consequential, as no observational data can validate the claimed formula. Measuring this failure requires queries that are provably non-identifiable, which prior evaluations lack, and grading that accepts correct formulas in any equivalent form, which string matching cannot provide. We build CERTID, a formal identification pipeline that addresses both limitations. CERTID uses the sound and complete causal identification algorithm ID to certify whether an effect is identifiable from a given graph and query, and verifies returned formulas against structural causal models whose interventional distributions are known exactly. CERTID further develops theoretical results to mitigate structural leakage, repair non-identifiable queries, and establish grading guarantees. We evaluate three frontier reasoning models (Gemini Flash, Gemini Pro, and GPT5.5) on 1,200 certified instances spanning 4 to 50 vertices. Accuracy proves a poor proxy for soundness: on identical instances, the false-claim rate on non-identifiable queries varies by seventeen-fold across models. We also find that models decide identifiability with 97-100% accuracy on graphs generated after the strongest model's training snapshot. Instances, the certification procedure, the verifier, and per-instance records are available at https://anonymous.4open.science/r/certid-D718.
Sources
- Experimental Design for Causal Effect Identification
- Reasoning Models Don't Always Say What They Think
- CogBench: a large language model walks into a psychology lab
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Machine Psychology
- Uncovering Hidden Correctness in LLM Causal Reasoning via Symbolic Verification
- Language Models (Mostly) Know What They Know
- Tulu 3: Pushing Frontiers in Open Language Model Post-Training
- EconCausal: A Context-Aware Economic Reasoning Benchmark for Large Language Models
- On causal inference with marked point process data
- CausalReasoningBenchmark: A Real-World Benchmark for Disentangled Evaluation of Causal Identification and Estimation
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- CASE: Causal Alignment and Structural Enforcement for Improving Chain-of-Thought Faithfulness
- LLM Cannot Discover Causality, and Should Be Restricted to Non-Decisional Support in Causal Discovery
- A Critical Review of Causal Reasoning Benchmarks for Large Language Models
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems