Measuring and Mitigating Solution Mode Collapse in RLVR
cs.LG, cs.CL
Submitted: 2026-10-08
Updated: 2026-10-09
License: http://creativecommons.org/licenses/by/4.0/
The gist: A language model (LM) can usually answer the same question in more than one way, but reinforcement learning with verifiable rewards (RLVR) is indifferent to which correct answer a model produces.
Terminology
Abstract
A language model (LM) can usually answer the same question in more than one way, but reinforcement learning with verifiable rewards (RLVR) is indifferent to which correct answer a model produces. A solution will earn the same reward whether it is the thousandth copy of a familiar answer or one the model has never produced before. Yet, there is potential value in having the model retain multiple correct solutions as it is trained. For instance, multiple modes may give users a choice and provide problem-solving strategies that improve overall model performance. Here, we introduce ModeBench, a benchmark of multi-solution tasks in which the verifier returns both correctness and mode discovered. We then use ModeBench to measure how solution diversity changes under RLVR post-training. We find that RLVR post-training concentrates probability onto fewer correct modes even as accuracy holds or improves, and moreover, that frontier models are already highly concentrated. We then introduce our solution, Re:Max, which stores one verified example per discovered mode in a replay buffer and trains on those stored modes uniformly. A solution found once is, therefore, practiced as often as one found repeatedly. Across three model scales, two RL objectives, and harder task constructions, replay improves both how often a policy succeeds and how many different ways it can succeed.
Sources
- How LLMs Distort Our Written Language
- SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model
- Large Language Monkeys: Scaling Inference Compute with Repeated Sampling
- Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models
- Training Verifiers to Solve Math Word Problems
- The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models
- RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment
- Argument Collapse: LLMs Flatten Long-Form Public Debate
- SetPO: Set-Level Policy Optimization for Diversity-Preserving LLM Reasoning
- The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Reward
- Beyond Mode Collapse: Distribution Matching for Diverse Reasoning
- Understanding R1-Zero-Like Training: A Critical Perspective
- Uniform-Correct Policy Optimization: Breaking RLVR's Indifference to Diversity
- Playing Atari with Deep Reinforcement Learning
- Illuminating search spaces by mapping elites
- Qwen2.5 Technical Report
- Experience Replay for Continual Learning
- Expected Return Causes Outcome-Level Mode Collapse in Reinforcement Learning and How to Fix It with Inverse Probability Scaling
- Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
- Outcome-based Exploration for LLM Reasoning
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks