Do Agent Benchmarks Do What They Say? An Executable-Contract Audit of Tool-Using Agent Environments
cs.SE, cs.AI, cs.LG
Submitted: 2026-09-29
Updated: 2026-09-29
Terminology
Sources
- LiveClawBench: Benchmarking LLM Agents on Complex, Real-World Assistant Tasks
- Automated Benchmark Auditing for AI Agents and Large Language Models
- Benchmarking the Benchmarks: A Validity Audit of Tool-Calling Evaluation
- Who Tests the Testers? Systematic Enumeration and Coverage Audit of LLM Agent Tool Call Safety
- MedAgentBench: A Realistic Virtual EHR Environment to Benchmark Medical LLM Agents
Related papers
- Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits
- GitSkills: A Dataset of Agent Skills on GitHub
- SABER: Benchmarking Operational Safety of LLM Coding Agents in Stateful Project Workspaces
- PackMonitor: Enabling Zero Package Hallucinations Through Decoding-Time Monitoring
- IntentCoding: Amplifying User Intent in Code Generation
- Incentives and Outcomes in Bug Bounties