AgentRx: Diagnosing AI Agent Failures from Execution Trajectories
cs.AI
Submitted: 2026-02-02
Updated: 2026-08-31
Comments: Accepted to EMNLP Findings 2026
License: http://creativecommons.org/licenses/by/4.0/
The gist: AI agents often fail in ways that are difficult to localize because executions are probabilistic, long-horizon, multi-agent, and mediated by noisy tool outputs.
Terminology
Abstract
AI agents often fail in ways that are difficult to localize because executions are probabilistic, long-horizon, multi-agent, and mediated by noisy tool outputs. We address this gap by manually annotating failed agent runs and release a novel benchmark of 170 trajectories across 11 diverse task settings, including structured API workflows, incident management, and open-ended web/file tasks. Each trajectory is annotated with a critical failure step and a category from a grounded-theory derived, cross-domain failure taxonomy. To mitigate the human cost of failure attribution, we present AgentRx, an automated diagnostic framework that pinpoints the critical failure step in a failed agent trajectory. It synthesizes constraints, evaluates them step-by-step, and produces an auditable validation log of constraint violations with associated evidence; an LLM-based judge uses this log to localize the critical step and category. AgentRx improves step localization by 75% on average over prior work, while providing failure category attribution.
Sources
- CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing
- AgentBench: Evaluating LLMs as Agents
- $\tau^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment
- Self-Refine: Iterative Refinement with Self-Feedback
- WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?
- $\tau$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
- Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems
- WebArena: A Realistic Web Environment for Building Autonomous Agents
- Agent-as-a-Judge: Evaluate Agents with Agents
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems