The Invisible Leash: Why RLVR May or May Not Escape Its Origin

arXiv:2507.14843 · cs.LG, cs.AI, cs.CL · Submitted 2025-07-20 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "The Invisible Leash".

Jane: The gist The current practice of RLVR primarily preserves rather than expands the base model’s solution coverage,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: Moving on to the title and authors of "The Invisible Leash: Why RLVR May or May Not Escape Its Origin," we see it's about testing if Reinforcement Learning with Verifiable Rewards is just preserving existing knowledge or actually expanding the model’s reasoning capability.

Jane: Fang Wu, Weihao Xuan, Ximing Lu, Mingjie Liu, Yi Dong, Zaid Harchaoui and Yejin Choi are the authors on this paper. It’s a solid group of researchers working in AI research.

Lu: Lu is involved in model training and representation learning at Tsinghua University. The paper seems to be pushing the idea that current RLVR practices might be too restrictive for true discovery.

Meng: From my side, I'm looking at this as a cautionary tale about relying too heavily on reward signals that only amplify what's already there, which is what I worry about when shipping models to production.

Tom: So the main point is that we need to figure out if RLVR is a tool for finding new things or just a fancy way of polishing old things.

Jane: Right, and the authors are specifically investigating this limitation by examining how RLVR can function as a support-constrained optimization mechanism, which restricts discovery to patterns already known by the base model.

Lu: They are also formalizing this by defining empirical support as the set of correct solutions that a model can realistically discover under finite sampling.

Meng: That’s important because it gives a concrete way to measure what's possible versus what's just noise in our sampling process.

Tom: So they are using these metrics, like Support Retention Rate and Net Discovery Rate, to track this loss of new solutions over time.

Jane: They are trying to determine if the gains we see in single-sample accuracy translate into actual expansion of the model's reasoning boundaries or if it's just sharper exploitation of existing heuristics.

The paper's summary: Tom: In this paper, they summarize the core idea is that RLVR might be amplifying high-reward outputs that the base model already knows instead of pushing the model to discover entirely new solutions.

Jane: They explain that RLVR involves optimizing a pretrained base model via Reinforcement Learning using simple, automatically computed rewards, and they formalize this by defining empirical support as those correct solutions discovered under finite sampling.

Lu: The key mechanism they formalize is the optimization of a pretrained base model via Reinforcement Learning using simple, automatically computed rewards to learn a new distribution.

Meng: So it boils down to taking an existing model and tweaking its parameters based on rewards, but the paper is focusing intensely on the nature of those rewards and what they unlock.

Tom: They set up this framework by defining the four regions based on whether a correct completion gets non-negligible probability mass from both the base model, q, and the policy theta, which leads to Support Preservation covering completions where both models have a probability greater than epsilon.

Jane: That means they are trying to categorize outcomes into regions based on whether both models can realistically generate them.

Lu: This categorization helps them map out exactly where solutions fall relative to the base model’s initial knowledge and where the RLVR process is operating.

Meng: It gives a clear map of where we are likely going—is it staying within the known good zones or heading somewhere new.

Tom: The empirical findings show that support preservation dominates across all domains when examining those support dynamics metrics, showing that even with improvements in single-sample accuracy, the shrinkage of empirical support generally outweighs the expansion of empirical support under larger sampling budgets.

Jane: That means the shrinkage of potential solutions is bigger than any gains we see from increasing our sampling budget.

The paper's improvements: Tom: Now for what they suggest as improvements, they aren't suggesting a complete overhaul but rather augmenting RLVR with explicit exploration or hybrid strategies that seed probability mass into underrepresented regions of the solution space.

Jane: They are suggesting we need to actively introduce mechanisms that deliberately push the model into areas where it hasn't learned anything yet.

Lu: This is a direct response to the "invisible leash" idea; it’s about breaking that constraint by seeding probability mass into underrepresented regions of the solution space.

Meng: I like the idea of hybrid strategies because we can control where those new samples are introduced, instead of just letting random exploration happen.

Tom: So we need to deliberately seed that exploration, not rely on standard RLVR to do it organically.

Jane: They also discuss decoupling local uncertainty from global diversity by analyzing token-level versus answer-level entropy to distinguish between stepwise uncertainty and overall support expansion.

Lu: This is a way of saying we can separate the process of getting slightly more certain about a step from the actual expansion of the solution space.

Meng: So, if token-level entropy goes up but answer-level entropy doesn't, we know we just have more noise in our immediate steps, not that we're finding fundamentally new final answers.

Tom: That’s a way to manage expectations about what the model is actually doing during training.

Jane: And theoretically, they propose using the variational framework to implement a policy update that explicitly minimizes KL divergence to the base distribution while satisfying those reward constraints.

Lu: This means we can design an update that respects both the known good solutions and pushes us toward new areas simultaneously by satisfying reward conditions while keeping things close to the original model structure.

Meng: That sounds like a structured way to manage that exploration, which is much better than just hoping for the best in standard RLVR.

Conclusion: Tom: To wrap up, the paper "The Invisible Leash: Why RLVR May or May Not Escape Its Origin" suggests that current RLVR primarily acts as a support-bounded optimizer, enhancing precision by concentrating mass on known high-reward solutions but rarely discovering new reasoning paths.

Jane: It boils down to saying it improves sampling efficiency by focusing probability on high-reward outputs, but this comes at the cost of reduced entropy and narrower exploration of the solution space.

Lu: The authors conclude that RLVR improves sampling efficiency in terms of concentration, but they emphasize that this is achieved at the expense of reduced entropy and narrower exploration for new reasoning structures.

Meng: So for us in production, we need to be careful not to get stuck optimizing for high reward and losing the ability to find solutions that are just slightly outside our immediate training distribution.

Lalam: I agree with that—the paper shows that the core trade-off is between precision gains and global diversity in the solution space.

Tom: So, while RLVR is efficient at refining existing solutions, we need to consider how to explicitly add exploration strategies if we want real expansion of reasoning boundaries.

Fang Wu, Weihao Xuan, Ximing Lu, Mingjie Liu, Yi Dong, Zaid Harchaoui, Yejin Choi

Stanford University · University of Tokyo · RIKEN AIP, Tokyo, Japan · University of Washington, Seattle, WA, USA

cs.LG, cs.AI, cs.CL

Submitted: 2025-07-20

Updated: 2026-10-14

Code: https://github.com/huggingface/lighteval

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 81/100

The gist: The gist The current practice of RLVR primarily preserves rather than expands the base model’s solution coverage, acting as a support-constrained optimization mechanism that may restrict discovery

Key concepts

Empirical Support: This refers to the set of correct solutions that a model can realistically discover given finite sampling budgets. It helps categorize outcomes based on whether a solution is likely to be found by both the base model and the RL policy, allowing researchers to measure how much accessible knowledge is preserved or lost.
Support Preservation vs. Expansion: The core finding shows that RLVR consistently loses access to more correct solutions than it gains. This means the optimization mechanism acts as a support-constrained filter, favoring known successes over discovering entirely new reasoning structures, effectively preserving existing knowledge rather than expanding the model's reachable solution space.
Entropy Trade-off: RLVR models often show increased average token-level entropy but reduced final answer entropy. This indicates that while the model becomes more uncertain about individual word choices, its final answers become more concentrated around a few specific solutions. This suggests a shift towards sharpening outputs on known modes rather than exploring diverse reasoning paths.

Terminology

Reinforcement Learning — 1× in this paper · explained in 4 episodes
Reinforcement Learning with Verifiable Rewards — 1× in this paper
Verifiable Rewards — 1× in this paper · explained in 2 episodes

Summary

The gist The current practice of RLVR primarily preserves rather than expands the base model’s solution coverage, acting as a support-constrained optimization mechanism that may restrict discovery to patterns already known by the base model<ref:2507.14843#pg20>

How it works

RLVR is investigated as a method for enhancing LLMs’ capabilities, but the study examines whether it expands reasoning boundaries or merely amplifies high-reward outputs that the base model already knows<ref:2507.14843#pg2> The core mechanism of RLVR involves optimizing a pretrained base model via Reinforcement Learning using simple, automatically computed rewards<ref:2507.14843#pg3> The study formalizes this by defining empirical support as the set of correct solutions that a model can realistically discover under finite sampling<ref:2507.14843#pg4> This framework allows for the categorization of outcomes into four regions based on whether a correct completion is assigned non-negligible probability mass by both models, q and πθ Support Preservation covers completions with q(y∗x) > ϵ and πθ(y∗x) > ϵ<ref:2507.14843#pg5>

Empirical Findings on Solution Accessibility

The empirical investigation reveals several key dynamics regarding solution accessibility under RLVR<ref:2507.14843#pg6> The study shows that RLVR consistently loses access to more correct solutions than it gains, even while improving single-sample accuracy across diverse reasoning benchmarks<ref:2507.14843#pg7> This is quantified by metrics such as Support Retention Rate (SRR), Net Discovery Rate (NDR), and the Support Dynamic Score (SDS)<ref:2507.14843#pg8> The results demonstrate that Support preservation dominates across all domains when examining support dynamics metrics<ref:2507.14843#pg9>

Trade-offs in Precision and Diversity

A fundamental finding is the precision-diversity trade-off, which is not domain-specific but reflects inherent properties of current RLVR<ref:2507.14843#pg10> The study notes that RLVR consistently improves avg@32 across benchmarks, raising average performance from 54.5% to 65.4% for ProRL and from 43.0% to 61.3% for DAPO but this increased precision comes at a cost: "RLVR systematically reduces answerlevel entropy, indicating a collapse onto fewer distinct solutions and empirically validating our theoretical prediction that reward optimization sharpens output distributions around known modes <ref:2507.14843#pg10> Furthermore, the analysis shows that answer-level entropy consistently declines while token-level entropy exhibits more varied behavior<ref:2507.14843#pg10> This phenomenon is termed local stochasticity without global exploration," where increased token-level entropy does not reliably predict the diversity of final solutions<ref:2507.14843#pg10>

Theoretical Limits and Constraints

Theoretical analysis confirms that RLVR is fundamentally constrained to operate within the support of the base model’s distribution<ref:2507.14843#pg10> Theorem C.1 proves that supp(πθ(· x)) ⊆ supp(q(· x)), meaning any correct solution y∗ with q(y∗x) = 0 remains inaccessible to policy gradient updates <ref:2507.14843#pg10> This is further formalized by the variational objective, which shows that the optimal policy takes the form π∗(y x) ∝ q(y x) · exp(βR(x, y))<ref:2507.14843#pg10> In the KL-free limit (β → ∞), the RLVR update converges to the renormalized restriction of the base model to the correct completion set <ref:2507.14843#pg10>

Conclusion and Future Directions

The overall takeaway is that RLVR acts as a support-bounded optimizer, enhancing precision by concentrating mass on known high-reward solutions but rarely discovering new reasoning paths<ref:2507.14843#pg10> To break this invisible leash, the paper suggests that RLVR may need to be augmented with explicit exploration or hybrid strategies that seed probability mass into underrepresented regions of the solution space <ref:2507.14843#pg10> This highlights a critical trade-off between precision gains and global diversity, necessitating mechanisms that deliberately seed probability mass into new reasoning structures<ref:2507.14843#pg10> The findings suggest that entropy reduction is not inherently undesirable but must be balanced with explicit exploration strategies to complement standard RLVR<ref:2507.14843#pg10> The paper concludes that RLVR improves sampling efficiency by concentrating probability on high-reward outputs, but this comes at the cost of reduced entropy and narrower exploration of the solution space<ref:2507.14843#pg10>

--- Page 26 ---

The invisible leash The current practice of RLVR primarily preserves rather than expands the base model’s solution coverage, acting as a support-constrained optimization mechanism that may restrict discovery to patterns already known by the base model<ref:2507.14843#pg10>

Theoretical Limits and Constraints

Improvements for AI systems

Bold header: Integration of Explicit Exploration Mechanisms

The improved system can be augmented with explicit exploration or hybrid strategies that seed probability mass into underrepresented regions of the solution space, moving beyond standard RLVR to discover solutions missed by the base model, as suggested in the conclusion.

Bold header: Dynamic Sampling Budget Adaptation

The system should implement a mechanism to adjust sampling budgets based on empirical support dynamics, using the shrinkage of empirical support generally outweighs the expansion of empirical support under larger sampling budgets to guide when to increase or decrease samples.

Bold header: Decoupling Local Uncertainty from Global Diversity

The system can be designed to distinguish between stepwise uncertainty and overall support expansion by analyzing token-level versus answer-level entropy, preventing the model from converging onto a smaller set of distinct answers simply because of increased local stochasticity.

Bold header: Support-Constrained Policy Projection

By utilizing the variational framework, the system can implement a policy update that explicitly minimizes KL divergence to the base distribution while satisfying reward constraints, effectively executing the optimal policy takes the form: π∗(y x) ∝ q(y x) · exp(βR(x, y)).

Bold header: Format-Aware Answer Processing

The answer extraction protocol must be enhanced with a hierarchical, priority-based extraction mechanism that specifically addresses task-specific formatting challenges, such as standardizing LATEX for fractions in dice tasks or removing whitespace from palindrome generation responses to ensure fair and objective accuracy assessments.

Abstract

Recent advances highlight Reinforcement Learning with Verifiable Rewards (RLVR) as a promising approach to enhancing LLM capabilities. However, it remains unclear whether the current practice of RLVR truly expands models' reasoning boundary or mainly amplifies high-reward outputs that base models already know. We introduce empirical support dynamics, a framework that quantifies how RLVR preserves, expands, and shrinks access to verified solutions. Applied across models, reasoning tasks, and training checkpoints, it matches verified successes question by question to reveal changes that aggregate accuracy can conceal. Across the evaluated model-level benchmark aggregates, the shrinkage of observed support outweighs its expansion: more questions lose a verified success than gain one at the stated sampling budgets. These losses occur alongside gains in pass@1. Some RLVR models increase average token-level entropy while reducing final-answer entropy: their token choices become more uncertain on average even as their final answers become more concentrated. Together, these findings reveal an ``invisible leash'' in current RLVR practice: it can make existing successes more reliable while losing access to other verified solutions. Empirical support dynamics make this imbalance measurable and motivate exploration strategies that reach underrepresented solutions while retaining prior successes.