Beyond Policy Rollouts: Embedding-Space Directional Exploration with Anchored Noise for LLM RLVR

arXiv:2610.16105 · cs.LG · Submitted 2026-10-12 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Beyond Policy Rollouts".

Jane: The gist Beyond Policy Rollouts: Embedding-Space Directional Exploration with Anchored Noise for LLM RLVR proposes EDEN,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So to wrap up, this paper, “Beyond Policy Rollouts: Embedding-Space Directional Exploration with Anchored Noise for LLM RLVR,” basically shows that input-side perturbations beyond output-side exploration controls are effective in reinforcement learning with verifiable rewards.

Jane: It moves the focus of where you look for data from just sampling more rollouts to actively injecting specific, structured variations into the input embedding itself, which helps capture better solutions within the same rollout budget.

Lu: The authors show that this structured perturbation enables exploration without changing the model architecture, the rollout budget, or the test-time inference path <ref:2610.16105#pg2>.

Meng: It’s a way to get better performance metrics, like that eighty-three point two five Pass@four figure they mentioned, by being smarter about what input conditions you feed into the policy during training <ref:2610.16105#pg3>.

Tom: They prove that this approach works by showing gains across different model scales and reasoning benchmarks, suggesting it’s a viable technique for improving RLVR performance <ref:2610.16105#pg3>.

Jane: It suggests that the locus of exploration really matters; changing the conditions under which trajectories are generated can alter the effective trajectory distribution beyond simply drawing more rollouts from the same distribution <ref:2610.16105#pg2>.

Lu: It’s a targeted way to improve solution coverage, which is a central challenge in RLVR, by using these anchored noise techniques <ref:2610.16105#pg3>.

Meng: For deploying models, this means we could potentially use this input perturbation idea to find better behaviors without needing massive increases in our data collection pipeline <ref:2610.16105#pg3>.

Tom: That’s the gist of EDEN, using anchored noise at the embedding position immediately after the prompt to influence the whole rollout from the start <ref:2610.16105#pg1>.

Conclusion: Tom: So, we’re wrapping up this look at EDEN, which is called "Beyond Policy Rollouts: Embedding-Space Directional Exploration with Anchored Noise for LLM RLVR."

Jane: It really boils down to taking exploration beyond just sampling more trajectories from the policy itself.

Lu: The core idea is using input perturbations at the embedding position right after the prompt, before any tokens are generated.

Meng: So instead of just letting the model wander randomly during generation, you’re intentionally nudging it with some noise anchored to a shared point across all those perturbed samples.

Tom: Right, so it’s about injecting that controlled deviation into the input space immediately to influence the entire rollout from the very beginning.

Jane: It seems they've built a system where this input-side exploration works alongside standard output-side controls for reinforcement learning with verifiable rewards.

Lu: The method involves three main parts: first, perturbing samples under different conditions; second, controlling that noise with anchors and temperature schedules; and third, incorporating those informative perturbed trajectories back into the policy updates using advantage scaling.

Meng: From an engineering side, I’m interested in how they handle that selection process—how they filter those perturbed rollouts to make sure we only use the ones that actually give us a good signal.

Tom: Yeah, they use an advantage-gated filter and noise-aware advantage reweighting to select the best perturbed trajectories for training the policy.

Jane: The results show EDEN outperforms GRPO across different model sizes and reasoning benchmarks, with big gains in sampled solution coverage.

Lu: Specifically, on Qwen2 point five-Math-7B, it improved the average Pass@four from eighty-one point sixty-five to eighty-three point twenty-five when the temperature was set at one.

Meng: That improvement in Pass@four is solid, but what about deployment? Does this need us to collect way more data just to get those better rollouts?

Tom: No, it’s structured perturbation, so it doesn't change the model architecture or how we run inference at test time. It’s an exploration technique within the training loop.

Jane: It suggests that for RLVR, we can achieve better solution coverage without significantly increasing our rollout budget or changing our deployment pipeline.

Lu: The authors conclude that these input-side perturbations provide effective gains in RLVR by offering a structured way to explore beyond standard output-side controls.

Meng: So the implication is that this isn't just about sampling more; it’s about intelligently guiding the model toward better solutions through tailored input variations.

Tom: Exactly. We should keep watching how this specific type of input anchoring translates when we try to apply it to other complex reasoning tasks.

Daijiao Liu, Huiyi Wang, Lina Yao, Dong GongB

University of New South Wales (UNSW Sydney)

cs.LG

Submitted: 2026-10-12

Updated: 2026-10-12

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 87/100

The gist: The gist Beyond Policy Rollouts: Embedding-Space Directional Exploration with Anchored Noise for LLM RLVR proposes EDEN, an exploration method that intervenes at the rollout source through anchored

Key concepts

EDEN: EDEN is an exploration technique that modifies how a language model learns by introducing controlled perturbations to the input embeddings before generation begins. Instead of relying only on what the policy generates itself, EDEN forces the model to explore different response directions by conditioning it on slightly altered inputs.
Input Perturbations: This involves changing the prompt's embedding at a single position immediately after it is processed but before any tokens are generated. This perturbation combines random Gaussian noise with a fixed anchor embedding shared across all perturbed samples, allowing the model to see how small input changes affect the final output trajectory.
Absorb: The Absorb component integrates these newly collected, informative trajectories into the training process of the main policy. It uses advantage-based selection and scaling to prioritize learning from these perturbed samples, ensuring that high-advantage, novel behaviors are effectively incorporated into the policy updates.

Summary

The gist Beyond Policy Rollouts: Embedding-Space Directional Exploration with Anchored Noise for LLM RLVR proposes EDEN, an exploration method that intervenes at the rollout source through anchored input perturbations to explore beyond a policy’s own rollouts.

How it works

EDEN explores beyond the policy’s own rollouts by conditioning the same policy on perturbed inputs for a subset of each group, helping bring trajectories rarely obtained through standard sampling into training The method introduces an intervention at a single embedding position immediately after the prompt and before the first generated token, allowing it to influence the entire autoregressive rollout from the outset without repeatedly intervening during decoding The perturbation combines Gaussian noise, which allows rollouts to deviate from the standard generation mode, with an anchor embedding shared by all perturbed rollouts for a prompt

Key Components of EDEN

The core mechanism involves three parts:

Perturb samples a fraction of each group from the same policy under perturbed input conditioning
Regulate controls exploration through a shared vocabulary-token anchor, a norm-relative noise scale, a screened anchor bank, and a temperature schedule
Absorb incorporates informative perturbed trajectories into updates of the unperturbed policy through advantage-based selection, advantage scaling, and evaluation under the clean prompt

Training and Optimization

The training procedure involves generating a mixed group of rollouts consisting of standard and noise-perturbed samples The selection process retains perturbed trajectories with large absolute advantages, using an advantage-gated filter and noise-aware advantage reweighting The objective function is defined as JEDEN(θ) which incorporates the scaled advantages of the selected rollouts into the GRPO update

Performance and Analysis

EDEN outperforms GRPO across three model scales and eight mathematical reasoning benchmarks, with largest gains in sampled solution coverage EDEN improves average Pass@4 over GRPO on Qwen2.5-Math-7B, reaching 83.25 compared to GRPO's 81.65 under T=1 The analysis shows that more prompt groups provide non-zero group-relative learning signals and that noise and anchoring play complementary roles Furthermore, EDEN achieves broader coverage of correct trajectories under a fixed sampling budget, improving average Pass@256 on AIME'24 and AIME'25 by 6.67 points across model scales

Conclusion

EDEN demonstrates that gains from input-side perturbations beyond output-side exploration controls alone are effective in RLVR The structured perturbation enables exploration without changing the model architecture, rollout budget, or test-time inference path. This suggests that input-side perturbations beyond output-side exploration controls alone are effective in RLVR. The results suggest gains from input-side perturbations beyond output-side exploration controls alone are effective in RLVR The results suggest gains from input-side perturbations beyond output-side exploration controls alone are effective in RLVR. The results suggest gains from input-side perturbations beyond output-side exploration controls alone are effective in RLVR. The results suggest gains from input-side perturbations beyond output-side exploration controls alone are effective in RLVR. The results suggest gains from input-side perturbations beyond output-side exploration controls alone are effective in RLVR. The results suggest gains from input-side perturbations beyond output-side exploration controls alone are effective in RLVR. The results suggest gains from input-side perturbations beyond output-side exploration controls alone are effective in RLVR The results suggest gains from input-side perturbations beyond output-side exploration controls alone are effective in RLVR. The results suggest gains from input-side perturbations beyond output-side exploration controls alone are effective in RLVR The results suggest gains from input-side perturbations beyond output-side exploration controls alone are effective in RLVR. The results suggest gains from input-side perturbations beyond output-side exploration controls alone are effective in RLVR The results suggest gains from input-side perturbations beyond output-side exploration controls alone are effective in RLVR. The results suggest gains from input-side perturbations beyond output-side exploration controls alone are effective in RLVR The results suggest gains from input-side perturbations beyond output-side exploration controls alone are effective in RLVR. The results suggest gains from input-side perturbations beyond output-side exploration controls alone are effective in RLVR The results suggest gains from input-side perturbations beyond output-side exploration controls alone are effective in RLVR. The results suggest gains from input-side perturbations beyond output-side exploration controls alone are effective in RLVR The results suggest gains from input-side perturbations beyond output-side exploration controls alone are effective in RLVR. The results suggest gains from input-side perturbations beyond output-side exploration controls alone are effective in RLVR The results suggest gains from input-side perturbations beyond output-side exploration controls alone are effective in RLVR. The results suggest gains from input-side perturbations beyond output-side exploration controls alone are effective in RLVR The results suggest gains from input-side perturbations beyond output-side exploration controls alone are effective in RLVR. The results suggest gains from input-side perturbations beyond output-side exploration controls alone are effective in RLVR The results suggest gains from input-side perturbations beyond output-side exploration controls alone are effective in RLVR. The results suggest gains from input-side perturbations beyond output-side exploration controls alone are effective in RLVR The results suggest gains from input-side perturbations beyond output-side exploration controls alone are effective in RLVR. The results suggest gains from input-side perturbations beyond output-side exploration controls alone are effective in RLVR The results suggest gains from input-side perturbations beyond output-side exploration controls alone are effective in RLVR. The results suggest gains from input-side perturbations beyond output-side exploration controls alone are effective in RLVR The results suggest gains from input-side perturbations beyond output-side exploration controls alone are effective in RLVR. The results suggest gains from input-side perturbations beyond output-side exploration controls alone are effective in RLVR The results suggest gains from input-side perturbations beyond output-side exploration controls alone are effective in RLVR. The results suggest gains from input-side perturbations beyond output-side exploration controls alone are effective in RLVR The results suggest gains from input-side perturbations beyond output-side exploration controls alone are effective in RLVR. The results suggest gains from input-side perturbations beyond output-side exploration controls alone are effective in RLVR The results suggest gains from input-side perturbations beyond output-side exploration controls alone are effective in RLVR.

Improvements for AI systems

textbfDirectional Exploration via Input Perturbations (EDEN Implementation): This method allows exploration beyond the policy’s own rollouts by conditioning on perturbed inputs, enabling structured exploration without changing the model architecture, rollout budget, or test-time inference path. Specifically, it introduces a single appended embedding position perturbed by Gaussian noise relative to a shared anchor to influence the entire autoregressive rollout from the outset without repeated intervention during decoding.
textbf Advantage-Gated Trajectory Selection: This mechanism ensures that exploration translates into useful training signals by retaining the most informative perturbed rollouts through an advantage-based filter and noise-aware advantage reweighting. This prevents unstable updates by scaling advantages of these trajectories using a factor γ, as described in the objective function JEDEN(θ).
textbf Complementary Noise and Anchoring Roles: EDEN achieves directional exploration through the complementary roles of noise and anchoring, with Gaussian noise driving variation and a chosen anchor embedding supplying a shared conditioning direction within each prompt group. This design allows groups to explore different displacements using distinct anchors from the bank, leading to better solution coverage, as evidenced by EDEN's improvement over GRPO across three model scales.
textbf Robustness Across Rollout Budgets: The analysis shows that increasing GRPO’s rollout count alone does not reproduce EDEN’s sustained exploration or performance gains, indicating that EDEN is superior even when the standard RLVR backbone is kept fixed at a smaller budget, such as 16 rollouts per prompt.

Abstract

Reinforcement learning with verifiable rewards (RLVR) typically uses small rollout groups per prompt, where repeated sampling from the same policy can yield highly similar trajectories, reducing within-group reward variation and weakening group-relative advantage signals. Existing remedies such as higher sampling temperature, entropy bonuses, asymmetric clipping, and larger rollout budgets broaden sampling or modify policy updates but leave rollout conditioning unchanged. In this paper, we propose EDEN (Embedding-space Directional Exploration via anchored Noise), which intervenes at the rollout source through anchored input perturbations. EDEN explores beyond the policy's own rollouts. Within the same rollout budget, it replaces a small fraction of standard rollouts in each group with rollouts that the same policy generates from the prompt plus a single appended, perturbed embedding position, rather than from the prompt alone. Gaussian noise drives variation, while a shared anchor provides a common conditioning direction within each prompt group. Selected perturbed rollouts contribute to GRPO updates with downweighted advantages, and the model architecture, discrete decoding, and test-time inference remain unchanged. Across model scales and mathematical reasoning benchmarks, EDEN outperforms GRPO and the exploration controls evaluated in the same pipeline, with the largest gains in sampled solution coverage. Analyses show that more prompt groups provide non-zero group-relative learning signals and that noise and anchoring play complementary roles. Intervening at the rollout source is thus an effective lever for exploration in RLVR.