Improving Reasoning Ability via Asynchronous On-Policy Self-Distillation under Positive Rollouts
Positive-Only Policy Optimization (POPO) is a reinforcement learning method that improves policy using only positive rollouts, avoiding negative rollouts entirely. It achieves stability and implicit penalties for incorrect answers through bounded importance sampling weights and entropy regularization. This approach yields performance comparable to or better than existing methods on mathematical reasoning benchmarks.