When KL Regularization Misfires in Group Policy Optimization
cs.LG, cs.CL
Submitted: 2026-10-08
Updated: 2026-10-08
Comments: 27 pages
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
The gist: Why does removing reference-policy KL regularization sometimes improve group policy optimization? This motivates studying how reference-policy information should enter group-relative updates.
Terminology
Abstract
Why does removing reference-policy KL regularization sometimes improve group policy optimization? This motivates studying how reference-policy information should enter group-relative updates. We analyze seven potential failure modes in the interactions between KL and rewards: residual KL updates after reward clipping, after gradient cancellation, and in groups with identical rewards; KL growth with response length and an imbalance in its relative contribution; KL concentration on a small number of tokens; and sampling noise when k1 is incorporated into rewards. We propose Zero-Sum Calibrated Policy Optimization (ZCPO), which uses relative drift measured by conditional KL to calibrate within-group reward coefficients and integrates them into the base surrogate. Mathematical reasoning experiments and ablations support this design's effectiveness in our settings.
Sources
- Beyond Euclidean Clipping: Overcoming Exploration Collapse in LLM RL via Riemannian Isometric Policy Optimization
- Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model
- AGPO: Adaptive Group Policy Optimization with Dual Statistical Feedback
- QUATRO: Query-Adaptive Trust Region Policy Optimization for LLM Fine-tuning
- GoLongRL: Capability-Oriented Long Context Reinforcement Learning with Multitask Alignment
- A Comedy of Estimators: On KL Regularization in RL Training of LLMs
- sGPO: Trading Inference FLOPs for Training Efficiency in RLVR
- On a few pitfalls in KL divergence gradient estimation for RL
- What is the Alignment Objective of GRPO?
- Anchored Policy Optimization: Mitigating Exploration Collapse Via Support-Constrained Rectification
- AGPO: Asymmetric Group Policy Optimization for Verifiable Reasoning and Search Ads Relevance at JD
- Group Sequence Policy Optimization
- Group Orthogonalized Policy Optimization:Group Policy Optimization as Orthogonal Projection in Hilbert Space
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks