Auditing LLM Benchmarks with Item Response Theory
cs.CL
Submitted: 2026-05-28
Updated: 2026-08-28
Comments: Accepted at EMNLP 2026. Associated data at https://huggingface.co/datasets/Writer/IRT-mislabeled-items
License: http://creativecommons.org/licenses/by-sa/4.0/
The gist: LLM benchmark labels are frozen at release and silently propagated into downstream benchmarks, errors and all.
Terminology
Abstract
LLM benchmark labels are frozen at release and silently propagated into downstream benchmarks, errors and all. We introduce an Item Response Theory-based indicator that surfaces likely mislabels at 95% precision in the top 200 examples across seven preference and multiple-choice benchmarks using responses from 114 models, outperforming a supervised classifier. We trace these errors to mechanical labeling heuristics, upstream annotation mistakes inherited unchanged from source datasets, and fundamentally ambiguous items without a defensible single label. The same model fit reveals that reward models specialize in stylistic preference rather than factual knowledge, and identifies one frontier reward model that agrees with detected mislabels at 78% accuracy versus 38% for its peers, consistent with benchmark contamination or benchmark-specific over-optimization.
Sources
- Training Verifiers to Solve Math Word Problems
- Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks
- Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy
- Do Large Language Model Benchmarks Test Reliability?
- Multiple-Choice Questions are Efficient and Robust LLM Evaluators
- Detecting Pretraining Data from Large Language Models
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering