Exploration-Preserving Policy Optimization
cs.AI
Submitted: 2026-10-02
Updated: 2026-10-02
Comments: 36 pages, 10 figures
Code: https://github.com/jinhangzhan/ExPPO
License: http://creativecommons.org/publicdomain/zero/1.0/
The gist: Reinforcement learning with verifiable rewards improves reasoning, while the allocation of learning signal shapes which solutions remain accessible under repeated sampling.
Abstract
Reinforcement learning with verifiable rewards improves reasoning, while the allocation of learning signal shapes which solutions remain accessible under repeated sampling. Group-relative objectives assign equal advantages to equally rewarded responses, making aggregate credit proportional to sampled mode frequency. We introduce Exploration-Preserving Policy Optimization (ExPPO), a lightweight advantage-shaping rule that redistributes credit using prompt-relative, length-normalized response surprisal and prompt pass rate. ExPPO combines bounded shaping with shared normalization to preserve verifier polarity and approximately maintain each prompt group's total absolute sequence-advantage mass. Our analysis characterizes response-level credit allocation alongside sampled mode updates, deriving local conditions for gains in entropy and correct-mode discovery. Experiments show improved in-domain and out-of-domain reasoning coverage, higher aggregate response accuracy, and strong coverage at large sampling budgets. A controlled multi-answer evaluation further demonstrates increased correct-mode yield and gains in diversity among verified-correct responses. Code is available at https://github.com/jinhangzhan/ExPPO
Sources
- Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
- When Correct Solutions Repeat: Rarity-Aware Credit Redistribution for GRPO
- Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models
- Group Entropy-Controlled Policy Optimization
- Maximum Entropy On-Policy Actor-Critic via Entropy Advantage Estimation
- The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models
- Differential Smoothing Mitigates Sharpening and Improves LLM Reasoning
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems
- Rewarding the Unlikely: Lifting GRPO Beyond Distribution Sharpening
- RL Fine-Tuning Heals OOD Forgetting in SFT
- Rotation-Preserving Supervised Fine-Tuning
- Solving Quantitative Reasoning Problems with Language Models
- Uniform-Correct Policy Optimization: Breaking RLVR's Indifference to Diversity
- Poly-EPO: Training Exploratory Reasoning Models
- Entropy-Preserving Reinforcement Learning
- F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rare
- Qwen2.5 Technical Report
- GPQA: A Graduate-Level Google-Proof Q&A Benchmark
- Qwen3 Technical Report
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems