Signals: Continuous Diagnostics of AI Agents
cs.AI
Submitted: 2026-11-03
Updated: 2026-11-04
Comments: 11 pages, 1 figure, 6 tables. Author affiliation formatting corrected; scientific content unchanged
Code: https://github.com/sierra-research/tau-bench
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
The gist: AI agents can make incorrect decisions or waste resources even when their tools and infrastructure work normally.
Terminology
Abstract
AI agents can make incorrect decisions or waste resources even when their tools and infrastructure work normally. Their nondeterministic behavior makes these problems difficult to reproduce and diagnose. We developed Signals to identify such problems from agentic traces and suggest concrete remediations. We built the method around a taxonomy of 45 problem types across eight families, with definitions and evidence requirements for each type. The method distinguishes realized failures, risk exposures, and inefficiencies, groups related findings into engineering issues called Signals, and assesses their severity from the supporting evidence. Each Signal includes actions a developer can take and checks for evaluating the change. Further executions can add evidence to an existing Signal or reveal a new problem. We evaluated the method on public agent executions and achieved 91.43% F1 when matching Signals to reference issues, about 18 percentage points higher than a one-pass LLM control that used the same model and taxonomy.
Sources
- $\tau$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
- Why Do Multi-Agent LLM Systems Fail?
- AgentRx: Diagnosing AI Agent Failures from Execution Trajectories
- RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models
- AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents
- SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
- Instruction-Following Evaluation for Large Language Models
- IHEval: Evaluating Language Models on Following the Instruction Hierarchy
- IH-Challenge: A Training Dataset to Improve Instruction Hierarchy on Frontier LLMs
- Demystifying the Lifecycle of Failures in Platform-Orchestrated Agentic Workflows
- ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities
- RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation
- Benchmarking Large Language Models in Retrieval-Augmented Generation
- CRAG -- Comprehensive RAG Benchmark
- HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal
- FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance
- RouterBench: A Benchmark for Multi-LLM Routing System
- AgentProcessBench: Diagnosing Step-Level Process Quality in Tool-Using Agents
- SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents
- Seeing the Whole Elephant: A Benchmark for Failure Attribution in LLM-based Multi-Agent Systems
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents