ThunderSyncRL: Lossless Acceleration of Agentic Reinforcement Learning
cs.LG, cs.AI
Submitted: 2026-10-05
Updated: 2026-10-05
License: http://creativecommons.org/licenses/by/4.0/
The gist: Language models are moving beyond generating answers to pursuing long-horizon goals in interactive environments.
Terminology
Abstract
Language models are moving beyond generating answers to pursuing long-horizon goals in interactive environments. Post-training these agents requires long, heterogeneous trajectories, and synchronous systems leave learner engines idle until rollout and verification finish. To squeeze out these pipeline bubbles, asynchronous training overlaps rollout and learning across updates, but comes at the cost of policy staleness. We introduce ThunderSyncRL, which starts gradient computation as soon as all required inputs are fixed, without policy staleness. For group relative policy optimization (GRPO), ThunderSyncRL computes each trajectory's score gradient as soon as the reward for that trajectory arrives, without waiting for the group. For on-policy distillation (OPD), it computes gradients for each completed agentic turn's teacher-scored actions while tool calls run in the sandbox. We prove that gradient streaming produces the same GRPO and OPD updates as batch-synchronous training, without changing either objective. On SWE-bench Verified and Terminal Bench 4.0, we train models to the same performance up to 1.9 times faster than synchronous training. With zero policy staleness, ThunderSyncRL also outperforms asynchronous training at a fixed budget by up to 2.47 percentage points.
Sources
- RolloutPipe: Overlapping Pipelined Rollout and Training in Disaggregated On-Policy LLM Reinforcement Learning
- Evaluating Large Language Models Trained on Code
- Libra: Efficient Resource Management for Agentic RL Post-Training
- RollPacker: Mitigating Long-Tail Rollouts for Fast, Synchronous RL Post-Training
- GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models
- Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning
- JigsawRL: Assembling RL Pipelines for Efficient LLM Post-Training
- AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM Post-Training
- TRACE: Capability-Targeted Agentic Training
- Agent Lightning: Train ANY AI Agents with Reinforcement Learning
- PipelineRL: Faster On-policy Reinforcement Learning for Long Sequence Generation
- TideRL: Boosting Agentic RL Goodput with Readiness-Aware Scheduling
- SETA: Scaling Environments for Terminal Agents
- R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Tree Training: Accelerating Agentic LLMs Training via Shared Prefix Reuse
- LlamaRL: A Distributed Asynchronous Reinforcement Learning Framework for Efficient Large-scale LLM Training
- RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning
- WAR: Workload-Aware Rollouts for Synchronous Agentic Reinforcement Learning
- DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scales
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks