AgentHop: A Diagnostic Benchmark for Agentic Multi-Hop Scientific Question Answering

arXiv:2609.34428 · cs.CL, cs.AI · Submitted 2026-09-28 · Read on arXiv

cs.CL, cs.AI

Submitted: 2026-09-28

Updated: 2026-09-28

Comments: Accepted to NeurIPS 2026 Evaluation and Datasets Track

Code: https://github.com/JohnnyNLP/agenthop

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Accuracy — 86× in this paper · explained in 1 episode
• Benchmark — 51× in this paper · explained in 2 episodes
• Ablation — 17× in this paper · explained in 1 episode
• RAG — 9× in this paper · explained in 3 episodes
• Robustness — 7× in this paper · explained in 5 episodes
• Agentic Task — 5× in this paper · explained in 1 episode
• Archetype — 5× in this paper · explained in 1 episode
• Core — 5× in this paper · explained in 2 episodes
• Affordance — 4× in this paper · explained in 1 episode
• Agentic Tasks — 4× in this paper · explained in 1 episode
• Benchmarking — 4× in this paper · explained in 3 episodes
• Failure Modes — 4× in this paper · explained in 3 episodes
• Grounding — 4× in this paper · explained in 3 episodes
• Structural Integrity — 4× in this paper · explained in 3 episodes
• Verification — 4× in this paper · explained in 2 episodes
• Agentic Evaluation — 3× in this paper · explained in 1 episode
• Consistency — 3× in this paper · explained in 3 episodes
• Reinforcement Learning — 3× in this paper · explained in 4 episodes
• Agentic Search — 2× in this paper · explained in 1 episode
• Alignment — 2× in this paper · explained in 1 episode
• Scaffolding — 2× in this paper · explained in 3 episodes
• Thinking mode — 2× in this paper · explained in 2 episodes
• Adam — 1× in this paper · explained in 2 episodes
• Agentic Loop — 1× in this paper · explained in 1 episode
• Agentic System — 1× in this paper · explained in 2 episodes
• Alignment Problem — 1× in this paper · explained in 1 episode
• Berkeley Function-Calling Leaderboard — 1× in this paper
• Fine-Tuning — 1× in this paper · explained in 5 episodes
• Gaussian Splatting — 1× in this paper · explained in 4 episodes
• Generative Model — 1× in this paper · explained in 2 episodes
• Knowledge Graph — 1× in this paper · explained in 7 episodes
• Knowledge graphs — 1× in this paper · explained in 2 episodes
• LLM judge — 1× in this paper · explained in 2 episodes
• Large Language Model (LLM) — 1× in this paper · explained in 3 episodes

Sources

• Cognitive Architectures for Language Agents
• ReAct: Synergizing Reasoning and Acting in Language Models
• AI Agents That Matter
• Measuring what Matters: Construct Validity in Large Language Model Benchmarks
• OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented LMs
• Towards Scientific Intelligence: A Survey of LLM-based Scientific Agents
• SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
• TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks
• TRAJECT-Bench:A Trajectory-Aware Benchmark for Evaluating Agentic Tool Use
• AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents
• HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
• Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps
• WebArena: A Realistic Web Environment for Building Autonomous Agents
• GAIA: a benchmark for General AI Assistants
• $\tau$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
• DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
• Kimi K2.5: Visual Agentic Intelligence
• Gemma 3 Technical Report
• Qwen3 Technical Report
• Reflexion: Language Agents with Verbal Reinforcement Learning

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering