Why Your Deep Research Agent Fails? On Hallucination Evaluation in Full Research Trajectory
cs.AI
Submitted: 2026-01-30
Updated: 2026-10-01
Code: https://github.com/yuhao-zhan/DeepHalluBench
License: http://creativecommons.org/licenses/by/4.0/
The gist: Diagnosing failure patterns in Deep Research Agents (DRAs) remains a critical challenge.
Terminology
Abstract
Diagnosing failure patterns in Deep Research Agents (DRAs) remains a critical challenge. Existing benchmarks predominantly rely on end-to-end evaluation, obscuring intermediate hallucinations that accumulate throughout the research trajectory. To bridge this gap, we propose a shift from outcome-based to process-aware evaluation by auditing hallucinations in the full plan-search-summarize trajectory. We introduce the PING Taxonomy, which categorizes DRA hallucinations into four complementary types: Propagation, Intent, Noise-induced, and Grounding. We further instantiate this taxonomy into a fine-grained evaluation framework that decomposes trajectories into atomic actions, claims, and sub-queries for rigorous verification, and we validate its reliability on standard fact-checking benchmarks and human-reviewed trajectories. Leveraging this framework to isolate 100 hallucination-prone tasks, including adversarial scenarios, we curate DeepHalluBench. Experiments on six representative DRAs show that, on our hallucination-prone stress-test set, all evaluated systems still exhibit non-negligible reliability gaps. Furthermore, our diagnostic analysis traces these failures to systemic deficits, especially hallucination propagation and cognitive biases, providing actionable insights for future architectural optimization. Code and data are available at https://github.com/yuhao-zhan/DeepHalluBench.
Sources
- DRBench: A Realistic Benchmark for Enterprise Deep Research
- Why Do Multi-Agent LLM Systems Fail?
- M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation
- xbench: Tracking Agents Productivity Scaling with Profession-Aligned Real-World Evaluations
- TRACE: Trajectory-Aware Comprehensive Evaluation for Deep Research Agents
- BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent
- DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
- Understanding DeepResearch via Reports
- IDRBench: Benchmarking the Interactive Capabilities of Deep Research Agents
- Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge
- DEER: A Benchmark for Evaluating Deep Research Agents on Expert Report Generation
- MMDeepResearch-Bench: A Benchmark for Multimodal Deep Research Agents
- DeepFact: Co-Evolving Benchmarks and Agents for Deep Research Factuality
- Deep Research Agents: A Systematic Examination And Roadmap
- Beyond the Final Answer: Evaluating the Reasoning Trajectories of Tool-Augmented Agents
- Llama2Vec: Unsupervised Adaptation of Large Language Models for Dense Retrieval
- ReportBench: Evaluating Deep Research Agents via Academic Survey Tasks
- DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Reports
- MM-BrowseComp: A Comprehensive Benchmark for Multimodal Browsing Agents
- Towards Personalized Deep Research: Benchmarks and Evaluations
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems