EasyPPO: Stabilizing the Critic Is Key
cs.LG
Submitted: 2026-09-29
Updated: 2026-09-29
Code: https://github.com/EasyPPO/EasyPPO
Project page: https://easyppo.github.io
Sources
- OpenAI o1 System Card
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Proximal Policy Optimization Algorithms
- FrontierCS: Evolving Challenges for Evolving Intelligence
- FrontierSmith: Synthesizing Open-Ended Coding Problems at Scale
- MLS-Bench: A Holistic and Rigorous Assessment of AI Systems on Building Better AI
- AutoLab: Can Frontier Models Solve Long-Horizon Auto Research and Engineering Tasks?
- EdgeBench: Unveiling Scaling Laws of Learning from Real-World Environments
- Nemotron-Cascade: Scaling Cascaded Reinforcement Learning for General-Purpose Reasoning Models
- Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning
- VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks
- Start Classifying: Categorical Critics for LLM Reinforcement Learning
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- When Do Larger Batches Help Scale LLM Reinforcement Learning?
- Tail-Likelihood Reinforcement Learning
- REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization
- JustRL: Scaling a 1.5B LLM with a Simple RL Recipe
- What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secret
- Best Practice Critic Optimization
- EVPO: Explained Variance Policy Optimization for Adaptive Critic Utilization in LLM Post-Training
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks