Agent Evaluation Reliability: More Tasks Won't (Always) Fix An Agent Leaderboard
cs.AI, cs.LG, stat.AP
Submitted: 2026-09-30
Updated: 2026-09-30
Code: https://github.com/hardy-education/scaffold_eval
Project page: https://www.kbb.com
Sources
- $\tau^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment
- Partial pooling predicts cross-validation reliability: a closed-form triage and Rao-Blackwellised cure for hierarchical LOO
- ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery
- Knowledge without Wisdom: Measuring Misalignment between LLMs and Intended Impact
- AI Cartography: Mapping the Latent Landscape of AI Benchmark Ecosystems
- AI Agents That Matter
- Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation
- Robustness in Large Language Models: A Survey of Mitigation Strategies and Evaluation Metrics
- Quantifying Variance in Evaluation Benchmarks
- Towards a Science of AI Agent Reliability
- Benchmarking Prompt Sensitivity in Large Language Models
- DISCO analysis: A nonparametric extension of analysis of variance
- Measurement to Meaning: A Validity-Centered Framework for AI Evaluation
- Harbor Adapters and Harbor-Index: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluation
- Can Language Models Solve Olympiad Programming?
- CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark
- AgentNoiseBench: Benchmarking Robustness of Tool-Using LLM Agents Under Noisy Condition
- An Illusion of Progress? Assessing the Current State of Web Agents
- $\tau$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
- WebArena: A Realistic Web Environment for Building Autonomous Agents
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems