FADE: Factored Ability--Difficulty Filtering for Online RLVR Scheduling
cs.LG
Submitted: 2026-11-02
Updated: 2026-11-04
License: http://creativecommons.org/licenses/by/4.0/
The gist: Group-based reinforcement learning with verifiable rewards (RLVR) learns only from prompts whose sampled completions disagree…
Terminology
Abstract
Group-based reinforcement learning with verifiable rewards (RLVR) learns only from prompts whose sampled completions disagree: all-pass and all-fail groups produce zero relative advantage and burn generation budget without moving the policy. Adaptive schedulers try to steer rollouts toward informative prompts, yet they rely on per-prompt pass-rate trackers that entangle two quantities evolving at very different speeds---the policy's shifting competence across reasoning skills and each prompt's intrinsic difficulty. When a single exponential average or Beta counter tracks both at once, short memory scrambles the relative ordering of prompts whereas long memory reports stale pass rates. We introduce FADE (Factored Ability--Difficulty Estimation), a bilinear state-space filter that separates the two timescales. FADE represents each prompt's success logit as the inner product of a whitened K-dimensional skill loading with a fast-drifting policy ability vector, minus a slow-shrinking scalar item intercept. Using Pólya--Gamma augmentation, each minibatch of binary rollout outcomes updates the K-dimensional ability posterior in closed form via a rank- K Woodbury step, instantly transferring competence shifts to all unobserved prompts that share those skills while leaving item intercepts stable. From the resulting posterior mean and variance we derive a closed-form utility for the probability of drawing a non-degenerate rollout group. Across five reasoning benchmarks, two base models, and four scheduling paradigms under a matched rollout budget, FADE consistently improves downstream accuracy and calibration while adding less than 0.6,ms per step.
Sources
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Kimi k1.5: Scaling Reinforcement Learning with LLMs
- Group Sequence Policy Optimization
- RLVR without Ineffective Samples: Group Prioritized Off-Policy Optimization for LLM Reasoning
- Dynamics-Predictive Sampling for Active RL Finetuning of Large Reasoning Models
- TRACE: A Unified Rollout Budget Allocation Framework for Efficient Agentic Reinforcement Learning
- Single-stream Policy Optimization
- Small Generalizable Prompt Predictive Models Can Steer Efficient RL Post-Training of Large Reasoning Models
- Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replay
- GRPO-LEAD: A Difficulty-Aware Reinforcement Learning Approach for Concise Mathematical Reasoning in Language Models
- Know When to Explore: Difficulty-Aware Certainty as a Guide for LLM Reinforcement Learning
- TACLer: Tailored Curriculum Reinforcement Learning for Efficient Reasoning
- CVPO: Enhancing LLM Reinforcement Learning Reasoning via Value-Variance Adaptation and Dynamic Curriculum Learning
- When Does Closed-Loop Curriculum Adaptation Help GRPO? An Analysis on Countdown
- Reinforce-Ada: An Adaptive Sampling Framework under Non-linear RL Objectives
- Learning as Reasoning Unfolds: Progressive Rollout Allocation for Efficient Reinforcement Learning
- SERA: Scale-Equalized Rollout Allocation for Maximum Likelihood Reinforcement Learning
- RollVerify: Bridging Efficiency and Accuracy in Long-Tail Rollout Reinforcement Learning
- GRPODropout: Less is More for Online Reinforcement Learning Rollouts
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis