Benchmarks Are Not Monolithic: Sample-Level Auditing and Orchestration for LLM Evaluation
Philipp D. Siedler, Jordan Sassoon
cs.CL, cs.AI, cs.LG
Submitted: 2026-07-30
Code: https://github.com/Aleph-Alpha-Research/eval-framework
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
The gist: Benchmark datasets are central to evaluating Large Language Models (LLMs), yet they are typically conceived as monolithic tasks, obscuring substantial variation in the demands of individual samples.
Terminology
Abstract
Benchmark datasets are central to evaluating Large Language Models (LLMs), yet they are typically conceived as monolithic tasks, obscuring substantial variation in the demands of individual samples. We introduce a dataset-centric meta-evaluation framework that audits benchmark datasets at the sample level along five latent dimensions: 1. Cognitive and Knowledge Demands, 2. Language and Content Quality, 3. Task Properties, 4. Context, and 5. Ethics, Safety, and Fairness. Applying this framework, we annotate five influential benchmarks -- MMLU, ARC, WinoGrande, HellaSwag, and TruthfulQA -- revealing pronounced internal heterogeneity that is not captured by aggregate accuracy scores. We show how these annotations enable criterion-driven orchestration of composite benchmark subsets across datasets, supporting targeted evaluation of model capabilities such as Reasoning Depth or Ethical Sensitivity. This approach reframes benchmark evaluation as dataset introspection, providing a principled methodology for analyzing and re-composing existing benchmarks to better reflect diverse evaluation needs.
Sources
- DeepSeek-V3 Technical Report
- Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
- Keeping Up with the Language Models: Systematic Benchmark Extension for Bias Auditing
- A large annotated corpus for learning natural language inference
- ERASER: A Benchmark to Evaluate Rationalized NLP Models
- Can Large Language Models be Trusted for Evaluation? Scalable Meta-Evaluation of LLMs as Evaluators via Agent Debate
- Who Benchmarks the Benchmarks? Towards Comprehensive Evaluation of Commonsense Reasoning Benchmarks
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- Annotation Artifacts in Natural Language Inference Data
- Measuring Massive Multitask Language Understanding
- Dynabench: Rethinking Benchmarking in NLP
- Prometheus: Inducing Fine-grained Evaluation Capability in Language Models
- RACE: Large-scale ReAding Comprehension Dataset From Examinations
- CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklists
- An Open Source Data Contamination Report for Large Language Models
- TruthfulQA: Measuring How Models Mimic Human Falsehoods
- G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
- The Llama 3 Herd of Models
- Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence
- The ConceptARC Benchmark: Evaluating Understanding and Generalization in the ARC Domain
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering