Calibration Is Not Enough: Evaluating Confidence Estimation Under Language Variations
cs.CL
Submitted: 2026-01-12
Updated: 2026-09-23
Code: https://github.com/vllm-project/vllm
License: http://creativecommons.org/licenses/by/4.0/
The gist: Confidence estimation (CE) indicates how reliable the answers of large language models are and impacts user trust and decision-making.
Terminology
Abstract
Confidence estimation (CE) indicates how reliable the answers of large language models are and impacts user trust and decision-making. Existing evaluations mainly concern the alignment between confidence and correctness, but ignore the variability of language: confidence estimates should remain consistent under semantically equivalent prompts or answer variations, while changing when answer meaning differs, as this may indicate a change in correctness. Therefore, we introduce a novel evaluation framework based on three complementary properties: robustness to prompt perturbations, stability across semantically equivalent answers, and sensitivity to semantically different answers. We show that these metrics are largely independent from existing CE metrics, and that common CE methods often fail on them: while most methods achieve high robustness and stability, they struggle to distinguish semantically different answers, potentially because they do not effectively leverage generation-side information. Overall, our framework exposes overlooked limitations of current CE evaluations and provides guidance for selecting confidence estimators for real-world applications.
Sources
- Uncertainty in Natural Language Generation: From Theory to Applications
- Language Model Cascades: Token-level uncertainty and beyond
- Reading Calibrated Uncertainty from Language Model Trajectories
- Language Models (Mostly) Know What They Know
- SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models
- 2 OLMo 2 Furious
- Anthropomimetic Uncertainty: What Verbalized Uncertainty in Language Models is Missing
- Calibrated Interpretation: Confidence Estimation in Semantic Parsing
- Revisiting Uncertainty Estimation and Calibration of Large Language Models
- Qwen2.5 Technical Report
- Black-box Uncertainty Quantification Method for LLM-as-a-Judge
- Towards Generation-Efficient Uncertainty Estimation in Large Language Models
- Can LLMs Assist Computer Education? an Empirical Case Study of DeepSeek
- Bayesian Reward Models for LLM Alignment
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering