Optimizing What Policies Learn From: Recoverability-aware Rollout Intervention Learning
Zheyuan Zhang, Manqing Mao, Hong Wang, Zhuoer Wang, Samson Koelle, Jie Yuan, Yanjun Lin, James Feng, Nikki Lijing Kuang, Yanfang Ye, Wei Niu
cs.LG, cs.CL
Submitted: 2026-08-05
License: http://creativecommons.org/licenses/by/4.0/
The gist: Critic-free group-based reinforcement learning has become a scalable approach for post-training large language models.
Terminology
Abstract
Critic-free group-based reinforcement learning has become a scalable approach for post-training large language models. However, most existing methods allocate the same number of rollouts to every task and trajectory state, even though some rollouts provide much more useful learning signals than others. Recent work has started to treat rollout generation as an adaptive decision, but two important limitations remain. First, intervention strategies are often based on fixed heuristics and therefore cannot adjust as the policy changes during training. Second, these methods usually decide only how many rollouts to generate, without explicitly controlling where and how to intervene. To address these limitations, we propose Recoverability-Aware Intervention Learning (RAIL), a training-time framework that learns how to generate rollouts based on the improvement produced by each intervention. RAIL models intervention selection as an online contextual-bandit problem and trains a recoverability controller using intervention traces collected through a shadow-to-live procedure. This allows the controller to keep learning while the underlying policy evolves. We evaluate RAIL in terms of effectiveness, adaptivity, expressiveness, and efficiency. Across multiple settings, RAIL consistently improves performance under limited rollout budgets. These results show that recoverability-aware intervention provides a principled way to generate more informative and less redundant rollouts, leading to stronger learning signals during post-training.
Sources
- Drift-Bench: Diagnosing Cooperative Breakdowns in LLM Agents under Input Faults via Multi-Turn Interaction
- Temperature as a Meta-Policy: Adaptive Temperature in LLM Reinforcement Learning
- Agentic Entropy-Balanced Policy Optimization
- Agentic Reinforced Policy Optimization
- UProp: Investigating the Uncertainty Propagation of LLMs in Multi-Step Agentic Decision-Making
- How to Allocate, How to Learn? Dynamic Rollout Allocation and Advantage Modulation for Policy Optimization
- 3SPO: State-Score-Supervised Policy Optimization for LLM Agents
- EvolveRouter: Co-Evolving Routing and Prompt for Multi-Agent Question Answering
- Tree Search for LLM Agent Reinforcement Learning
- Understanding R1-Zero-Like Training: A Critical Perspective
- Adaptive Rollout Allocation for Online Reinforcement Learning with Verifiable Rewards
- Group Distributionally Robust Optimization-Driven Reinforcement Learning for LLM Reasoning
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- SAGE: Answer-Conditioned Uncertainty Targets for Verbal Uncertainty Alignment
- RAGEN-2: Reasoning Collapse in Agentic RL
- Entropy-Tree: Tree-Based Decoding with Entropy-Guided Exploration
- Advancing the Pareto Frontier of Test-Time Scaling with Multi-Path Plan Aggregation
- Reinforce-Ada: An Adaptive Sampling Framework under Non-linear RL Objectives
- Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Exploration
- CoBA-RL: Capability-Oriented Budget Allocation for Reinforcement Learning in LLMs
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks