Beyond Post-Hoc Temperature Scaling: Bilevel Optimization for LLM Calibration
Ruochen Jin, Zhanliang Wang, Zongyu Dai, Jiancong Xiao, Bojian Hou
cs.LG
Submitted: 2026-08-07
Updated: 2026-08-10
Comments: Third Conference on Language Modeling (COLM 2026)
Code: https://github.com/BojianHou/calm
License: http://creativecommons.org/licenses/by/4.0/
The gist: Preference alignment often makes large language models (LLMs) overconfident and poorly calibrated.
Terminology
Abstract
Preference alignment often makes large language models (LLMs) overconfident and poorly calibrated. Traditional post-hoc temperature scaling is inherently domain-dependent: a temperature fitted on one domain does not generalize across domains. This motivates us to modify model parameters during training to improve calibration. We propose maximizing the entropy of predictive distributions as the calibration objective, which directly targets overconfidence by discouraging overly concentrated predictions. Inspired by temperature scaling, we realize this through a bilevel optimization formulation, where the lower level trains the model under a parametric loss and the upper level selects loss hyperparameters to maximize entropy. To make the framework practical at LLM scale, we adopt an efficient first-order approximation that avoids explicit second-order computation. Across both multiple-choice and open-ended generative question answering, experiments demonstrate that our method yields well-calibrated LLMs with particular advantages in out-of-domain generalization.
Sources
- Sparks of Artificial General Intelligence: Early experiments with GPT-4
- Dataset Reset Policy Optimization for RLHF
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- Learn Your Reference Model for Real Good Alignment
- Enhancing Confidence Expression in Large Language Models Through Learning from Past Experience
- Investigating Uncertainty Calibration of Aligned Language Models under the Multiple-Choice Setting
- Mistral 7B
- Language Models (Mostly) Know What They Know
- Tulu 3: Pushing Frontiers in Open Language Model Post-Training
- Policy Optimization in RLHF: The Impact of Out-of-preference Data
- Statistical Impossibility and Possibility of Aligning LLMs with Human Preferences: From Condorcet Paradox to Nash Equilibrium
- 2 OLMo 2 Furious
- GPT-4 Technical Report
- From $r$ to $Q^*$: Your Language Model is Secretly a Q-Function
- Proximal Policy Optimization Algorithms
- Fundamental Limits of Game-Theoretic LLM Alignment: Smith Consistency and Preference Matching
- Understanding the performance gap between online and offline alignment algorithms
- Gemini: A Family of Highly Capable Multimodal Models
- Llama 2: Open Foundation and Fine-Tuned Chat Models
- A Semantic-Sampling Framework for Evaluating Calibration in Open-Ended Question Answering
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks