Controlling Sharpening in LLM Post-Training
cs.LG
Submitted: 2026-11-02
Updated: 2026-11-02
License: http://creativecommons.org/licenses/by/4.0/
The gist: LLM post-training via reinforcement learning has shown consistent improvements in accuracy at the cost of reducing behavioral coverage - a phenomenon known as sharpening that can stall progress on…
Terminology
Abstract
LLM post-training via reinforcement learning has shown consistent improvements in accuracy at the cost of reducing behavioral coverage - a phenomenon known as sharpening that can stall progress on difficult tasks. Repeated sampling objectives such as pass@k and max@k are known to mitigate sharpening, but the mechanism behind it remains poorly understood. Moreover, existing approaches for@k optimization use dedicated algorithms, and no general framework exists for adapting established policy optimization methods from@1 to@k. We derive the optimal policies for KL-regularized pass@k and max@k objectives in closed form. Our explicit solutions reveal that@k optimization naturally redistributes learning pressure based on prompt difficulty, directing it toward harder problems. Crucially, we show that standard@1 objectives achieve these exact same optimal policies through prompt-adaptive regularization or reward transformation. Ultimately, this yields an optimizer-independent explanation for why@k objectives reduce sharpening on easy tasks while prioritizing success on harder ones. Building on these insights, we introduce@k Adaptive Pressure for Policy Optimization (KAPPO). Following a one-time reference calibration, KAPPO seamlessly adapts existing@1 methods to@k via prompt-adaptive regularization or reward transformation, without otherwise altering their training procedures. Experiments with several RL methods on mathematical reasoning and code generation demonstrate that KAPPO provides direct control over sharpening. Crucially, we demonstrate that@k pressure can be fully realized with single completion training, matching the performance of multi-completion@k optimizers at a fraction of the cost.
Sources
- Evaluating Large Language Models Trained on Code
- AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learning
- Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models
- POETS: Uncertainty-Aware LLM Optimization via Compute-Efficient Policy Ensembles
- OrderGrad: Optimizing Beyond the Mean with Order-Statistic Policy Gradient Estimation
- Beyond the Sampled Token: Preserving Candidate Support in RLVR
- Proximal Policy Optimization Algorithms
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Outcome-based Exploration for LLM Reasoning
- Olmo 3
- LeetCodeDataset: A Temporal Dataset for Robust Evaluation and Efficient Training of Code LLMs
- Qwen3 Technical Report
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis