Efficient Evaluation of LLM Performance with Statistical Guarantees

arXiv:2601.20251 · stat.ML, cs.LG · Submitted 2026-01-28 · Read on arXiv

stat.ML, cs.LG

Submitted: 2026-01-28

Updated: 2026-10-29

Comments: 31 pages, 12 figures

Code: https://github.com/skbwu/e

License: http://creativecommons.org/licenses/by/4.0/

The gist: Exhaustively evaluating many large language models (LLMs) on a large suite of benchmarks is expensive.

Terminology

Abstract

Exhaustively evaluating many large language models (LLMs) on a large suite of benchmarks is expensive. We cast benchmarking as finite-population inference and, under a fixed query budget, seek tight confidence intervals (CIs) for model accuracy with asymptotic frequentist coverage. We propose Factorized Active Querying (FAQ), which (a) leverages historical information through a Bayesian factor model; (b) adaptively selects questions using a hybrid variance-reduction/active-learning sampling policy; and (c) maintains validity through Pro-Active Inference---a finite-population extension of active inference (Zrnic & Candès, 2024) that enables direct question selection while preserving asymptotic frequentist coverage. In our experiments, FAQ delivers up to 5 times effective sample size gains over strong baselines on two benchmark suites, across varying historical-data missingness levels, with negligible computational overhead: this means that it matches the CI width of uniform sampling while using up to 5 times fewer queries. We release our source code and our curated datasets to support reproducibility and future research.

Sources

Related papers