Deconstructing Off-Policy Ratios: Entropy-Normalized Trust Regions for Asynchronous Reinforcement Learning
cs.AI
Submitted: 2026-07-24
Updated: 2026-10-08
License: http://creativecommons.org/licenses/by/4.0/
The gist: Asynchronous reinforcement learning (RL) accelerates large language model (LLM) post-training by overlapping rollout generation with policy optimization, but the resulting stale, off-policy data
Terminology
Abstract
Asynchronous reinforcement learning (RL) accelerates large language model (LLM) post-training by overlapping rollout generation with policy optimization, but the resulting stale, off-policy data destabilizes optimization and can cause policy collapse. Existing methods gate tokens by ratio magnitude alone, applying one threshold at every position. We show that the ratio's natural scale is set by token entropy, so deviations from mid-trajectory weight updates stay within this scale and carry genuine exploration. We further identify an overlooked low-entropy regime that breaks this scaling, where a near-zero probability amplifies train--inference mismatch into noise far beyond what the local entropy admits. A magnitude threshold admits this noise and discards the exploration. We therefore propose the Entropy-Normalized Trust Region (ENTR). Across long-horizon agentic tasks and mathematical reasoning benchmarks, ENTR outperforms existing asynchronous methods. It improves avg@1 on BrowseComp-Plus by 6.9% over the strongest baseline, trains stably up to 30 policy versions of staleness, and matches synchronous GRPO at a 2.6 times speedup.
Sources
- AsyncWebRL: Efficient Asynchronous Reinforcement Learning for Multi-Step Visual Web Agents
- BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent
- Training Verifiers to Solve Math Word Problems
- Group-in-Group Policy Optimization for LLM Agent Training
- AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning
- From $\log \pi$ to $\pi$: Taming Divergence in Soft Clipping via Bilateral Decoupled Decay of Probability Gradient Weight
- GLM-5: from Vibe Coding to Agentic Engineering
- Missing Old Logits in Asynchronous Agentic RL: Semantic Mismatch and Repair Methods for Off-Policy Correction
- AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM Post-Training
- Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning
- DORA: A Scalable Asynchronous Reinforcement Learning System for Language Model Training
- Stable Asynchrony: Variance-Controlled Off-Policy RL for LLMs
- Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning
- StaleFlow: Staleness-Aware Data Management for Mitigating Data Skewness in Fully Disaggregated RL Post-Training
- Trust Region Masking for Long-Horizon LLM Reinforcement Learning
- BandPO: Bridging Trust Regions and Ratio Clipping via Probability-Aware Bounds for LLM Reinforcement Learning
- ASymPO: Asymmetric-Scale Policy Optimization for Asynchronous LLM Post-Training Without Behavior Information
- AcceRL: A Distributed Asynchronous Reinforcement Learning and World Model Framework for Vision-Language-Action Models
- Part II: ROLL Flash -- Accelerating RLVR and Agentic Training with Asynchrony
- Ratio-Variance Regularized Policy Optimization for Efficient LLM Fine-tuning
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems