Learning at the Right Pace: Adaptive Data Scheduling Improves LLM Reinforcement Learning
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: Today's paper: "Learning at the Right Pace".
Jane: The gist The Adaptive Data Scheduling (ADS) framework, a dual-level data scheduling system,
Tom: First, who's behind it and why it matters.
Title and authors: Tom: The title itself, "Learning at the Right Pace: Adaptive Data Scheduling Improves LLM Reinforcement Learning," tells you that the core idea is about timing and matching the data to what the model can actually handle at that moment.
Jane: It’s about moving away from a fixed schedule where you just feed everything in evenly, toward a system that adjusts based on two things: what kind of patterns are in the data, and how capable the AI is right now.
Lu: The authors are taking this idea of adapting to semantic structure and policy capability and building it into a practical framework called Adaptive Data Scheduling, which they describe as having two levels of scheduling.
Meng: So instead of one static way to pick samples from the whole dataset, they are suggesting a more dynamic system that considers both the meaning of the prompt and where the model is currently struggling or succeeding.
Lalam: The goal is to replace that uniform sampling with something smarter, organizing data into groups based on shared patterns and then adapting how often we pull from those groups based on their success rate.
The paper's summary: Tom: They start by saying existing methods like GRPO, which uses relative outcome comparisons within a group, rely on a naive, unstructured data sampling strategy where you just maximize the expected loss over prompts drawn uniformly from the entire dataset.
Jane: That uniform distribution is what they find is the problem because it doesn't account for where samples lie relative to the policy’s current capability boundary, which leads to advantage collapse.
Lu: They propose organizing training data into semantic clusters based on shared patterns and meanings first, and then maintaining an adaptive inter-cluster distribution by updating the probability of selecting each cluster according to its cluster-level success rate.
Meng: So if a certain group of prompts—say, those about physics—is succeeding well, the system should start pulling more from that physics group before shifting focus elsewhere.
Lalam: And they also have intra-cluster scheduling which focuses on selecting policy-boundary samples within those clusters by tracking empirical success rates near fifty percent.
The paper's improvements: Tom: One of the key improvements they highlight is that this dual-level structure allows for inter-cluster distribution that consolidates progress at the cluster level, meaning it helps solidify learning on specific semantic patterns before moving to harder concepts.
Jane: And at the sample level, they focus on policy-boundary samples, which are those near where the model's current capability is shifting, because those rollout groups are likely to contain both correct and incorrect responses.
Lu: They define a difficulty score d i for each sample based on the negative log-likelihood of its reference solution under the base policy, and then they use an online rollout feedback rule to update a mini-cluster M m using samples near fifty percent success rate.
Meng: That sounds like it’s trying to avoid getting stuck with prompts that are either too easy or way too hard, which is something I worry about when we deploy models at scale.
Lalam: They show this approach works across different objective designs, meaning it's RL objective-agnostic and can improve methods with different reinforcement learning objectives like GRPO, OPD, DAPO, and GSPO.
Conclusion: Tom: So to wrap up the Adaptive Data Scheduling paper, they show that both the inter-cluster distribution and intra-cluster scheduling are necessary; removing either one makes the performance worse than using the full ADS framework.
Jane: They also showed that this dynamic pacing produces what they call a "spiral reward increase," which reflects how it continuously adapts to where the policy's capability boundary is shifting during training.
Lu: It’s important to remember that this offline preprocessing stage, which computes those sample embeddings and difficulty scores, only happens once before training starts and doesn't add extra per-step overhead during the actual policy optimization.
Meng: That upfront cost for preprocessing has to be manageable if we want to use this in a production environment where we're running things continuously.
Lalam: Overall, Learning at the Right Pace: Adaptive Data Scheduling Improves LLM Reinforcement Learning replaces uniform sampling with an adaptive distribution over semantic clusters and policy-boundary sample selection to pace RL posttraining according to both semantic structure and policy capability.
Tom: It’s a solid framework for making sure we are training on the most useful data available at any given point in the process.
Jane: It really changes how we think about pacing model learning after initial training, moving beyond just feeding it more data.
Johns Hopkins University · Rice University · University of Haifa · Workato · University of North Carolina at Charlotte
cs.CL
Submitted: 2026-06-21
Updated: 2026-10-14
Comments: 15 pages, 4 figures
Code: https://github.com/Richard-zrx/ADS
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 88/100
The gist: The gist The Adaptive Data Scheduling (ADS) framework, a dual-level data scheduling system, improves Large Language Model reinforcement learning by replacing uniform sampling with an adaptive
Key concepts
- Semantic Clustering
- This process organizes training data into groups based on shared patterns and meanings derived from the base policy's embeddings. It creates semantically coherent collections of prompts that share underlying conceptual structures, allowing the system to understand the relationships between different types of problems.
- Inter-Cluster Distribution
- ADS uses a formula to adapt how often samples are drawn from each semantic cluster during training. This distribution prioritizes clusters that have shown higher success rates so the policy focuses its learning efforts on areas where it is currently performing best, ensuring progress is made efficiently.
- Policy-Boundary Samples
- These are specific data points located near the current limits of what the model can achieve. By tracking samples near a 50% success rate within a cluster, ADS identifies informative examples that lie right at the edge of the policy's capability, helping it learn how to improve its performance in challenging areas.
Terminology
Summary
The gist The Adaptive Data Scheduling (ADS) framework, a dual-level data scheduling system, improves Large Language Model reinforcement learning by replacing uniform sampling with an adaptive distribution over semantic clusters and policy-boundary sample selection.
How it works
-
Semantic Clustering: ADS first organizes the training data into semantic clusters according to shared patterns and meanings using embeddings from the base policy to form semantically coherent groups.
-
Inter-Cluster Distribution: During training, ADS maintains an adaptive inter-cluster distribution by updating the probability of selecting each cluster based on its cluster-level success rate, where higher success rates are prioritized.
-
Intra-Cluster Scheduling: Within selected clusters, ADS performs intra-cluster scheduling to continuously sample policy-boundary samples by tracking empirical success rates near 50%.
Key Components and Mechanisms
The framework consists of three key modules: semantic clustering, inter-cluster sampling distribution, and intra-cluster scheduling. The goal is to pace RL posttraining according to both semantic structure and policy capability.
The inter-cluster distribution adapts the training schedule by using the formula p m+1 k = r m P k / K where r m is the cluster-level success rate. This mechanism allows the policy to consolidate progress on semantically related samples before shifting toward more challenging ones.
The intra-cluster scheduling focuses on selecting policy-boundary samples,
which are those near the current capability boundary where rollout groups likely contain both correct and incorrect responses. This is achieved by assigning an offline difficulty score di to each sample based on the negative log-likelihood of its reference solution under the base policy. The boundary mini-cluster Mm k is then updated using an online rollout feedback rule that replaces samples outside the [0.5 - epsilon, 0.5 + epsilon] success rate band with nearby alternatives. The gains are consistent across different objective designs, highlighting its potential as a general data scheduling strategy.
ADS also demonstrates strong OOD generalization beyond the mathematical reasoning domain used for training, showing gains in GPQA-Diamond and MMLU-Pro datasets. Furthermore, ADS achieves consistent improvements across models of different parameter scales, supporting its adaptive design.
Ablation Study Insights
Ablation studies confirm the necessity of both modules; removing either the inter-cluster distribution or intra-cluster scheduling leads to worse performance than the full ADS. The training dynamics analysis shows that ADS produces a spiral reward increase
reflecting its continuous adaptation to the policy’s capability boundary.
The offline preprocessing stage, which computes sample embeddings and difficulty scores, is performed only once before training starts and does not introduce additional per-step overhead during policy optimization.
Conclusion
In this work, Adaptive Data Scheduling (ADS) is introduced as a dual-level data scheduling framework that enhances LLM post-training by replacing standard uniform sampling with an adaptive distribution and policy-boundary sample selection. ADS enables the training policy to learn at an optimal pace by adapting sampling toward semantic clusters aligned with the policy’s current capability and selecting informative samples near the capability boundary. Experimental results show that ADS consistently improves accuracy and generalizes across different RL objectives.
--- Page 1 ---
The gist The Adaptive Data Scheduling (ADS) framework, a dual-level data scheduling system, improves Large Language Model reinforcement learning by replacing uniform sampling with an adaptive distribution over semantic clusters and policy-boundary sample selection.
The inter-cluster distribution adapts the training schedule by using the formula p m+1 k = r m P k / K where r m is the cluster-level success rate<ref:2606.
Improvements for AI systems
-
Adaptive data scheduling enables
the policy to consolidate progress on semantically related samples before gradually shifting toward more challenging ones in the training set.
This allows LLMs to solidify learning on specific semantic patterns before moving to harder concepts, preventing policy oscillation between unrelated patterns. -
Policy-boundary sample selection ensures that training is
not dominated by samples that have become too easy or remain too difficult
by focusing on samples whererollout groups are likely to contain both correct and incorrect responses.
This prevents advantage collapse caused by overly easy or overly difficult prompts, leading to more informative relative advantages. -
The dual-level scheduling structure allows for
inter-cluster distribution that consolidates progress
(cluster level) andintra-cluster scheduling that tracks informative samples
(sample level). This systematic pacing ensures the RL post-training process is guided by both semantic coherence and the policy's evolving capability boundary. -
The framework is
RL Objective-Agnostic,
meaning it canimprove methods with different RL objective designs,
demonstrating its robustness across various optimization strategies like GRPO, OPD, DAPO, and GSPO. This suggests the scheduling mechanism provides a general data distribution improvement applicable to many LLM RL post-training pipelines. -
The system can achieve
strong and stable accuracy gains in both GPQA-D and MMLU-Pro datasets
by generalizing beyond mathematical reasoning to scientific domains like biology, physics, and chemistry. This indicates the scheduling strategy strengthens general reasoning behaviors rather than merely overfitting to mathematical patterns.
Sources
- VeriThinker: Learning to Verify Makes Reasoning Model Efficient
- Olmo 3
- Understanding R1-Zero-Like Training: A Critical Perspective
- Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Models
- Clip-Low Increases Entropy and Clip-High Decreases Entropy in Reinforcement Learning of Large Language Models
- GPQA: A Graduate-Level Google-Proof Q&A Benchmark
- Proximal Policy Optimization Algorithms
- Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replay
- Qwen3 Technical Report
- DTS: Enhancing Large Reasoning Models via Decoding Tree Sketching
- Learning to Reason under Off-Policy Guidance
- Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement
- DAPO: An Open-Source LLM Reinforcement Learning System at Scale
- ExGRPO: Learning to Reason from Experience
- A Survey of Reinforcement Learning for Large Reasoning Models
- Group Sequence Policy Optimization
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering