Cool the Sampler, Not the Learner: Sampling Temperature Moves the Staleness Cliff of Importance-Corrected GRPO
cs.LG, cs.CL
Submitted: 2026-09-29
Updated: 2026-09-29
Comments: 14 pages, 8 figures, 4 tables
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Terminology
Sources
- Training Verifiers to Solve Math Word Problems
- Temperature as a Meta-Policy: Adaptive Temperature in LLM Reinforcement Learning
- AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning
- Stable Asynchrony: Variance-Controlled Off-Policy RL for LLMs
- StaleFlow: Staleness-Aware Data Management for Mitigating Data Skewness in Fully Disaggregated RL Post-Training
- A-3PO: Accelerating Asynchronous LLM Training with Staleness-aware Proximal Policy Approximation
- Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training
- Part II: ROLL Flash -- Accelerating RLVR and Agentic Training with Asynchrony
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- HybridFlow: A Flexible and Efficient RLHF Framework
- Scaling Laws for Collapse in Asynchronous GRPO
- How Off-Policy Can GRPO Be? Mu-GRPO for Efficient LLM Reinforcement Learning
- GAC: Stabilizing Asynchronous RL Training for LLMs via Gradient Alignment Control
- Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement
- Internalize the Temperature: On-Policy Self-Distillation as Policy Reheater for Reinforcement Learning
- Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?
- Look Inward to Explore Outward: Learning Temperature Policy from LLM Internal States via Hierarchical RL
- Exploring Multi-Temperature Strategies for Token- and Rollout-Level Control in RLVR
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks