Towards Reliable LLM Evaluation: Correcting the Winner's Curse in Adaptive Benchmarking
stat.ML, cs.AI, cs.LG, stat.AP
Submitted: 2026-05-07
Updated: 2026-11-07
Code: https://github.com/jznmsl/siren
Terminology
Sources
- Cost-Optimal Active AI Model Evaluation
- PaLM 2 Technical Report
- Optimal Inference After Model Selection
- A Framework for Efficient Model Evaluation through Stratification, Sampling, and Estimation
- The Llama 3 Herd of Models
- DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines
- GPT-4 Technical Report
- Efficient Evaluation of LLM Performance with Statistical Guarantees
- How Benchmark Prediction from Fewer Data Misses the Mark
- Efficient Evaluation of Large Language Models via Collaborative Filtering
Related papers
- Uniform-in-Time Weak Propagation-of-Chaos in Shallow Neural Networks
- Diffusion Models Observe Only Gradients: A Geometric Perspective on Score Matching Errors
- Quantifying Error Propagation and Model Collapse in Diffusion Models
- Behavior of prediction performance metrics with rare events
- Statistical Inference and Learning for Shapley Additive Explanations (SHAP)
- Control Variate Design for Score-Function Gradient Estimators in Importance-Weighted Variational Inference