Headroom-Drift Replay: A Primitive for Principled Replay Control in GRPO
cs.LG, cs.AI, cs.CL
Submitted: 2026-09-03
Updated: 2026-09-03
Comments: 51 pages, 25 figures, 17 tables. Accepted at COLM 2026
License: http://creativecommons.org/licenses/by/4.0/
The gist: RL-based post-training for reasoning models is increasingly bottlenecked by repeated fresh rollout generation, particularly in agentic settings where environment interaction dominates wall-clock cost.
Terminology
Abstract
RL-based post-training for reasoning models is increasingly bottlenecked by repeated fresh rollout generation, particularly in agentic settings where environment interaction dominates wall-clock cost. Replay can reduce this burden by reusing past trajectories, but existing methods typically embed it within larger training pipelines involving exploration, experience restructuring, or mixed-policy optimization. This makes replay's own contribution difficult to isolate. We ask a focused question: how far can principled replay selection alone go? We introduce Headroom-Drift Replay, a group-level replay control primitive for GRPO that separates reuse into two decisions. Headroom ranks stored groups by remaining learning value, while Drift gates them by compatibility with the current policy. The fresh on-policy stream remains unchanged, and the method adds no auxiliary generation or training machinery. Across mathematical reasoning, multimodal reasoning, and Agentic Search benchmarks, this single intervention outperforms naive replay and matches or exceeds broader replay methods on Avg Mean@32. In Agentic Search, where environment interaction dominates cost, it delivers comparable quality at materially lower wall-clock time.
Sources
- Constitutional AI: Harmlessness from AI Feedback
- RePO: Replay-Enhanced Policy Optimization
- Entropy-Preserving Reinforcement Learning
- Agentic Reinforcement Learning with Implicit Step Rewards
- MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention
- Prioritized Experience Replay
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- EFRame: Deeper Reasoning via Exploration-Filter-Replay Reinforcement Learning Framework
- Information Gain-based Policy Optimization: A Simple and Effective Approach for Multi-Turn Search Agents
- On the Entropy Dynamics in Reinforcement Fine-Tuning of Large Language Models
- ExGRPO: Learning to Reason from Experience
- Group Sequence Policy Optimization
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks