Where MLLMs Fail and Why: Causal Task Decomposition for Capability Failure Diagnosis
cs.CL, cs.AI, cs.CV, cs.LG
Submitted: 2026-09-30
Updated: 2026-09-30
Code: https://github.com/anuragxel/roadwork-dataset
License: http://creativecommons.org/licenses/by-sa/4.0/
The gist: End-to-end accuracy on compositional tasks records how often MLLMs fail, but cannot distinguish whether a failure reflects an intrinsic deficit in the targeted capability or a cascading error from an
Terminology
Abstract
End-to-end accuracy on compositional tasks records how often MLLMs fail, but cannot distinguish whether a failure reflects an intrinsic deficit in the targeted capability or a cascading error from an upstream prerequisite. We propose a causal decomposition framework that isolates these two failure modes through controlled interventions on the prerequisite dependencies of each task. Our capability metrics (NC, IC, RC) score each task under unassisted, correct, or incorrect prerequisites to diagnose where failures arise; contribution metrics (N-Score, S-Score), adapted from probabilities of causation, quantify each prerequisite's necessity and sufficiency to determine why. We instantiate the framework in CADET, a diagnostic benchmark of 10 composite tasks decomposed into 46 unit tasks with over 33,000 human-annotated questions spanning perception, spatial, temporal, and cognitive categories. Diagnosing frontier MLLMs with our framework uncovers systematic patterns that end-to-end accuracy obscures. Capability-wise, supplying correct prerequisites eliminates 54% of errors on cognitive tasks, lifting them from weakest to above spatial and temporal. Prerequisite-wise, causal contributions are concentrated in a few critical prerequisites, and supplying the single most important one alone captures 84% of the gain from supplying all prerequisites.
Sources
- CausalFlow: Causal Attribution and Counterfactual Repair for LLM Agent Failures
- BabyVision: Visual Reasoning Beyond Language
- Reasoning Models Don't Always Say What They Think
- Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmark
- Which Modality Decides? Counterfactual Modality Attribution for Multimodal LLMs
- Causal Agent Replay: Counterfactual Attribution for LLM-Agent Failures
- Kimi K2.5: Visual Agentic Intelligence
- From Flat Logs to Causal Graphs: Hierarchical Failure Attribution for LLM-based Multi-Agent Systems
- Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting
- Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering