Silent Instrument Errors Reorder Agent Leaderboards
cs.AI, cs.SE
Submitted: 2026-11-03
Updated: 2026-11-03
Comments: 12 pages, 1 figure, 2 tables. Code and reproduction scripts: https://github.com/YangzeLiu/eval-instrument-disclosure
Code: https://github.com/sierra-research/tau2-bench
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Terminology
Sources
- $\tau^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment
- AI Agents That Matter
- Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation
- Log analysis is necessary for credible evaluation of AI agents
- Asking Earns Nothing: Scoring the Decision to Act in BFCL Multi-Turn
- What Does a Harness Buy? Tokens, Mostly
- Rollout Cards: A Reproducibility Standard for Agent Research
- $\tau$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
- UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench
- Establishing Best Practices for Building Rigorous Agentic Benchmarks
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents