Stabilizing Policy Optimization via Logits Convexity
cs.LG, cs.CL
Submitted: 2026-03-01
Updated: 2026-11-07
Code: https://github.com/huggingface/trl
Terminology
Sources
- TLCR: Token-Level Continuous Reward for Fine-grained Reinforcement Learning from Human Feedback
- MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention
- Discriminative Policy Optimization for Token-Level Reward Models
- The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models
- MiniLLM: On-Policy Distillation of Large Language Models
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization
- Generalist Reward Models: Found Inside Large Language Models
- High-Dimensional Continuous Control Using Generalized Advantage Estimation
- Proximal Policy Optimization Algorithms
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Ring-lite: Scalable Reasoning via C3PO-Stabilized Reinforcement Learning for LLMs
- Qwen3 Technical Report
- DCPO: Dynamic Clipping Policy Optimization
- DAPO: An Open-Source LLM Reinforcement Learning System at Scale
- What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secret
- R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learning
- Group Sequence Policy Optimization
- DPO Meets PPO: Reinforced Token Optimization for RLHF
- CARFT: Boosting LLM Reasoning via Contrastive Learning with Annotated Chain-of-Thought-based Reinforced Fine-Tuning
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis