A Theory of Policy Optimization in Post-Training: Coverage Collapse and Reference Resets
cs.LG, stat.ML
Submitted: 2026-11-10
Updated: 2026-11-10
Terminology
Sources
- Global Optimality Guarantees For Policy Gradient Methods
- Large Language Monkeys: Scaling Inference Compute with Repeated Sampling
- Provable Benefit of Curriculum in Transformer Tree-Reasoning Post-Training
- Convex Optimization: Algorithms and Complexity
- KARL: Knowledge Agents via Reinforcement Learning
- The Coverage Principle: How Pre-Training Enables Post-Training
- Soft Policy Optimization: Online Off-Policy RL for Sequence Models
- The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models
- DeepSeek-V3 Technical Report
- Unlocking Reasoning Capabilities in LLMs via Reinforcement Learning Exploration
- Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO
- Is a Good Foundation Necessary for Efficient Reinforcement Learning? The Computational Role of the Base Model in Exploration
- REBEL: Reinforcement Learning via Regressing Relative Rewards
- A Theory of Regularized Markov Decision Processes
- The Llama 3 Herd of Models
- Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model
- Low-probability Tokens Sustain Exploration in Reinforcement Learning with Verifiable Reward
- The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarization
- On the Emergence of Implicit Curriculum in RLVR Learning Dynamics
- A Closer Look at Deep Policy Gradients
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis