What Drives the Zero-to-One Jump? Diagnosing and Steering Hard-Prompt Learning in RLVR
cs.LG, cs.AI
Submitted: 2026-11-02
Updated: 2026-11-02
Journal ref: Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 41504-41520, 2026
DOI: 10.18653/2026.emnlp-main.1821
Code: https://github.com/jameszhu-pixel-time/Z2O
License: http://creativecommons.org/licenses/by/4.0/
The gist: Reinforcement Learning with Verifiable Rewards (RLVR) has substantially improved mathematical reasoning, yet progress on hard prompts often stalls…
Terminology
Abstract
Reinforcement Learning with Verifiable Rewards (RLVR) has substantially improved mathematical reasoning, yet progress on hard prompts often stalls: a prompt can remain unsolved for many training steps before abruptly transitioning to solved--a discrete "zero-to-one" jump whose underlying dynamics and steerability remain poorly understood. We investigate two questions: what drives this transition, and can it be steered? Through event-based analysis measuring the time-to-first-positive (TTFP), we find that negative rollouts preceding successful transitions consistently retain higher early-stage generation entropy, distinguishing near-solvable failures from dead ends. This signal motivates NAPO (Negative-Augmented Policy Optimization), a method integrated with Group Relative Policy Optimization (GRPO) that ranks negatives in unsolved groups via an early-branching driver and assigns graded advantages, injecting informative gradients precisely where standard group-relative updates vanish. Under fixed rollout budgets, NAPO accelerates early solution discovery and improves Pass@256 by 10.0 points on AIME24 and 6.6 points on AIME2025, demonstrating that zero-to-one transitions are diagnosable and steerable through driver-guided negative learning. Our code is available at https://github.com/jameszhu-pixel-time/Z2O.
Sources
- Evaluating Large Language Models Trained on Code
- How to Allocate, How to Learn? Dynamic Rollout Allocation and Advantage Modulation for Policy Optimization
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
- LIMR: Less is More for RL Scaling
- TSPO: Breaking the Double Homogenization Dilemma in Multi-turn Search Policy Optimization
- MATH-Beyond: A Benchmark for RL to Expand Beyond the Base Model
- NGRPO: Negative-enhanced Group Relative Policy Optimization
- Proximal Policy Optimization Algorithms
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Qwen3 Technical Report
- Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement
- Let it Calm: Exploratory Annealed Decoding for Verifiable Reinforcement Learning
- LLM Probability Concentration: How Alignment Shrinks the Generative Horizon
- Group Sequence Policy Optimization
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis