Who Guards the Benchmarks? Automated Auditing of LLM Agent Benchmarks
BENCHGUARD is an automated auditing framework that systematically checks agent benchmarks for flaws using large language models as auditors. It cross-verifies instructions, environments, and solutions through structured reasoning to find systemic logical errors invisible to humans. This means benchmarks can be more reliable by ensuring they measure what they claim to measure.