Agent Benchmarks Stop Ranking Before They Run Out of Tasks
cs.AI
Submitted: 2026-11-10
Updated: 2026-11-10
Terminology
Sources
- When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation
- SWE-Bench+: Enhanced Coding Benchmark for LLMs
- CADWorld: Computer-Use Benchmark for Long-Horizon Computer-Aided Design
- How Benchmarks Mis-Score Computer-Use Agents
- Cost-of-Pass: An Economic Framework for Evaluating Language Models
- The Benchmark Ceiling: Human Judgment, Evaluation Scarcity, and the Political Economy of AI Capability Measurement
- Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation
- AI Agents That Matter
- Towards Robust Agentic CUDA Kernel Benchmarking, Verification, and Optimization
- Coding Agents Have Converged: Why the SWE-bench Leaderboard Can No Longer Order Its Top Entries, and What to Measure Instead
- Quantifying Variance in Evaluation Benchmarks
- Benchmarking at the Edge of Comprehension
- Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations
- OpenComputer: Verifiable Software Worlds for Computer-Use Agents
- SWE-ABS: Adversarial Benchmark Strengthening Exposes Inflated Success Rates on Test-based Benchmark
- When Verifiable Counts Depend on Wording: Auditing Wording Robustness in Instruction Following
- One Token to Fool LLM-as-a-Judge
- Establishing Best Practices for Building Rigorous Agentic Benchmarks
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents