Holdout Best-of-N: Unbiased Evaluation and Its Cost
cs.CL
Submitted: 2026-10-06
Updated: 2026-10-06
Sources
- Evaluating Large Language Models Trained on Code
- Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking
- Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignment
- Inference-Time Reward Hacking in Large Language Models
- Let's Verify Step by Step
- Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations
- WebGPT: Browser-assisted question-answering with human feedback
- Soft Best-of-n Sampling for Model Alignment
- Agent-as-a-Judge: Evaluate Agents with Agents
- Fine-Tuning Language Models from Human Preferences
- A Flexible Defense Against the Winner's Curse
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering