Pessimism's Paradox: Conservative Offline Training Amplifies Reward Hacking During Online Adaptation in Reasoning Models
Subramanyam Sahoo, Aman Chadha, Vinija Jain, Divya Chaudhary
cs.LG, cs.AI, stat.ML
Submitted: 2026-06-29
Comments: Accepted in ICML 2026 workshop on Decision-Making from Offline Datasets to Online Adaptation: Black-Box Optimization to Reinforcement Learning
Code: https://github.com/huggingface/trl
License: http://creativecommons.org/licenses/by/4.0/
The gist: Conservative offline training is widely advocated as a safe foundation for subsequent online adaptation: if a policy stays close to well-supported behaviour, the argument goes, it is less likely to
Terminology
Abstract
Conservative offline training is widely advocated as a safe foundation for subsequent online adaptation: if a policy stays close to well-supported behaviour, the argument goes, it is less likely to exploit imperfections in a learned reward model. We challenge this intuition empirically and mechanistically. We train a Qwen3-14B policy under Direct Preference Optimisation (DPO) with three levels of conservatism (beta in beta lo, beta mid, beta hi derived from empirical log-ratio percentiles), then adapt each checkpoint online against a learned reward ensemble (3, times,Qwen3-1.7B) while measuring true performance on GSM8K exact-answer accuracy. We find that higher offline conservatism monotonically increases reward-hacking damage, measured by the Goodhart gap and its area under the curve (AUGC), with Spearman rho = 1.0 across all three conditions. Mechanistic analysis reveals a three-link causal chain: (i) high- beta DPO compresses policy entropy, (ii) Low-entropy policies generate responses with reduced diversity, concentrating in a narrow region of the reward model's training distribution (lower pairwise cosine distance), and (iii) despite this proximity, ensemble disagreement (epistemic uncertainty) increases with beta and is exploited faster during online optimisation. We further fit a power-law curve to the (beta,) data and identify a practical optimal conservatism level beta that balances alignment fidelity against hacking vulnerability. Our results suggest that the field needs calibrated, not maximal, conservatism.
Sources
- Offline Reinforcement Learning with Implicit Q-Learning
- Conservative Q-Learning for Offline Reinforcement Learning
- Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
- Decision Transformer: Reinforcement Learning via Sequence Modeling
- Understanding plasticity in neural networks
- The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Defining and Characterizing Reward Hacking
- UltraFeedback: Boosting Language Models with Scaled AI Feedback
- QLoRA: Efficient Finetuning of Quantized LLMs
- Scaling Laws for Reward Model Overoptimization
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks