Diversifying RLVR Rollouts via First-Token Exploration
The paper addresses a diversity bottleneck in Reinforcement Learning with Verifiable Rewards (RLVR) by targeting low-load, high-leverage positions in the reasoning trace. It found that diversifying the very first token after a reasoning marker significantly improves RLVR performance without changing other pipeline components. This intervention reallocates rollout budget toward plausible initial routes, leading to better coverage and correctness.