What Drives the Zero-to-One Jump? Diagnosing and Steering Hard-Prompt Learning in RLVR

arXiv:2611.02747 · cs.LG, cs.AI · Submitted 2026-11-02 · Read on arXiv

cs.LG, cs.AI

Submitted: 2026-11-02

Updated: 2026-11-02

Journal ref: Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 41504-41520, 2026

DOI: 10.18653/2026.emnlp-main.1821

Code: https://github.com/jameszhu-pixel-time/Z2O

License: http://creativecommons.org/licenses/by/4.0/

The gist: Reinforcement Learning with Verifiable Rewards (RLVR) has substantially improved mathematical reasoning, yet progress on hard prompts often stalls…

Terminology

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has substantially improved mathematical reasoning, yet progress on hard prompts often stalls: a prompt can remain unsolved for many training steps before abruptly transitioning to solved--a discrete "zero-to-one" jump whose underlying dynamics and steerability remain poorly understood. We investigate two questions: what drives this transition, and can it be steered? Through event-based analysis measuring the time-to-first-positive (TTFP), we find that negative rollouts preceding successful transitions consistently retain higher early-stage generation entropy, distinguishing near-solvable failures from dead ends. This signal motivates NAPO (Negative-Augmented Policy Optimization), a method integrated with Group Relative Policy Optimization (GRPO) that ranks negatives in unsolved groups via an early-branching driver and assigns graded advantages, injecting informative gradients precisely where standard group-relative updates vanish. Under fixed rollout budgets, NAPO accelerates early solution discovery and improves Pass@256 by 10.0 points on AIME24 and 6.6 points on AIME2025, demonstrating that zero-to-one transitions are diagnosable and steerable through driver-guided negative learning. Our code is available at https://github.com/jameszhu-pixel-time/Z2O.

Sources

Related papers