On Calibration of Large Language Models: From Response To Capability
This work introduces capability calibration to fix a flaw in LLM confidence estimation. Response calibration measures single output correctness, but capability calibration targets the model's expected accuracy across all possible outputs. The framework defines this target and proposes methods to estimate it, showing that this approach leads to better performance in tasks like pass@k prediction.