Efficient Evaluation of LLM Performance with Statistical Guarantees
stat.ML, cs.LG
Submitted: 2026-01-28
Updated: 2026-10-29
Comments: 31 pages, 12 figures
Code: https://github.com/skbwu/e
License: http://creativecommons.org/licenses/by/4.0/
The gist: Exhaustively evaluating many large language models (LLMs) on a large suite of benchmarks is expensive.
Terminology
Abstract
Exhaustively evaluating many large language models (LLMs) on a large suite of benchmarks is expensive. We cast benchmarking as finite-population inference and, under a fixed query budget, seek tight confidence intervals (CIs) for model accuracy with asymptotic frequentist coverage. We propose Factorized Active Querying (FAQ), which (a) leverages historical information through a Bayesian factor model; (b) adaptively selects questions using a hybrid variance-reduction/active-learning sampling policy; and (c) maintains validity through Pro-Active Inference---a finite-population extension of active inference (Zrnic & Candès, 2024) that enables direct question selection while preserving asymptotic frequentist coverage. In our experiments, FAQ delivers up to 5 times effective sample size gains over strong baselines on two benchmark suites, across varying historical-data missingness levels, with negligible computational overhead: this means that it matches the CI width of uniform sampling while using up to 5 times fewer queries. We release our source code and our curated datasets to support reproducibility and future research.
Sources
- A Survey on Large Language Models for Critical Societal Domains: Finance, Healthcare, and Law
- GPT-4 Technical Report
- Anatomy of a Machine Learning Ecosystem: 2 Million Models on Hugging Face
- Preference Leakage: A Contamination Problem in LLM-as-a-judge
- Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Following
- LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks
- LongReasonArena: A Long Reasoning Benchmark for Large Language Models
- Active Evaluation Acquisition for Efficient LLM Benchmarking
- Active Learning and Bayesian Optimization: a Unified Perspective to Learn with a Goal
- How Benchmark Prediction from Fewer Data Misses the Mark
- tinyBenchmarks: evaluating LLMs with fewer examples
- Efficient Evaluation of Large Language Models via Collaborative Filtering
- Cost-Optimal Active AI Model Evaluation
- Open-LLM-Leaderboard: From Multi-choice to Open-style Questions for LLMs Evaluation, Benchmark, and Arena
Related papers
- Uniform-in-Time Weak Propagation-of-Chaos in Shallow Neural Networks
- Diffusion Models Observe Only Gradients: A Geometric Perspective on Score Matching Errors
- Quantifying Error Propagation and Model Collapse in Diffusion Models
- Behavior of prediction performance metrics with rare events
- Statistical Inference and Learning for Shapley Additive Explanations (SHAP)
- Control Variate Design for Score-Function Gradient Estimators in Importance-Weighted Variational Inference