Your Group-Relative Advantage Is Biased
summary
The gist
Reinforcement Learning from Verifier Rewards (RLVR) methods like Group-Relative Policy Optimization (GRPO) rely on group-relative advantage estimation which is inherently biased relative to the true…
In short
Group-Relative Policy Optimization (GRPO) methods in Reinforcement Learning from Verifier Rewards use group-relative advantages to train large language models, but this estimation is systematically biased. The bias causes underestimation for hard prompts and overestimation for easy ones. The authors propose History-Aware Adaptive Difficulty Weighting (HA-DW), an adaptive reweighting scheme that adjusts advantage estimates based on evolving difficulty anchors and training dynamics. This correction improves model performance across benchmarks.
Key concepts
- Group-Relative Advantage Estimation
- This is a method used in RL where the advantage of taking an action is estimated by comparing it to the average outcome of a group of similar actions. In this context, it measures how much better or worse a specific prompt response is compared to other responses within its group, which guides policy updates.
- Conditional Estimation Bias
- This refers to the systematic error in advantage estimation that depends on the specific condition or difficulty of the input. The analysis showed that under binary rewards, this bias is not random but predictable: it consistently underestimates advantages for hard prompts and overestimates them for easy prompts, especially when prompt difficulty deviates from 0.5.
- History-Aware Adaptive Difficulty Weighting (HA-DW)
- This is the proposed solution. HA-DW uses two parts: an evolving difficulty anchor that tracks long-term reward trends from past batches, and an adaptive reweighting algorithm. This mechanism dynamically adjusts advantage estimates using this history to compensate for the inherent bias in the initial estimation.
- Difficulty Anchor
- This is a component of HA-DW that incorporates cross-batch historical information into a buffer. It captures long-term reward trends by propagating the model's prior through this history, helping to establish a more stable baseline for assessing prompt difficulty over time.
Terminology used across episodes
This episode discusses
- Your Group-Relative Advantage Is Biased · Paper Radio
- LSPO: Length-aware Dynamic Sampling for Policy Optimization in LLM Reasoning
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- FAPO: Flawed-Aware Policy Optimization for Efficient and Reliable Reasoning
- A domain-theoretic framework for conditional probability and Bayesian updating in programming
- Segment Policy Optimization: Effective Segment-Level Credit Assignment in RL for Large Language Models
- MAPO: Mixed Advantage Policy Optimization
- SPEC-RL: Accelerating On-Policy Reinforcement Learning with Speculative Rollouts
- Understanding R1-Zero-Like Training: A Critical Perspective
- Proximal Policy Optimization Algorithms
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- IntentionReasoner: Facilitating Adaptive LLM Safeguards through Intent Reasoning and Selective Query Refinement
- HybridFlow: A Flexible and Efficient RLHF Framework
- KTAE: A Model-Free Algorithm to Key-Tokens Advantage Estimation in Mathematical Reasoning
- Scaling Behaviors of LLM Reinforcement Learning Post-Training: An Empirical Study in Mathematical Reasoning
- Qwen3 Technical Report
- DaGRPO: Rectifying Gradient Conflict in Reasoning via Distinctiveness-Aware Group Relative Policy Optimization
- Reinforce-Ada: An Adaptive Sampling Framework under Non-linear RL Objectives
- DCPO: Dynamic Clipping Policy Optimization
- DAPO: An Open-Source LLM Reinforcement Learning System at Scale
- Learning Like Humans: Advancing LLM Reasoning Capabilities via Adaptive Difficulty Curriculum Learning and Expert-Guided Self-Reformulation
The paper
Your Group-Relative Advantage Is Biased · Read on arXiv
Fengkai Yang, Zherui Chen, Xiaohan Wang, Xiaodong Lu, Jiajun Chai, Guojun Yin, Wei Lin, Shuai Ma, Fuzhen Zhuang, Deqing Wang, Yaodong Yang
Beihang University 2025 University of California Berkeley 2025 Peking University 4 Meituan
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Your Group-Relative Advantage Is Biased".
Jane: Here's the gist of the paper. Reinforcement Learning from Verifier Rewards (RLVR) methods like Group-Relative Policy Optimization (GRPO) rely on group-relative advantage estimation which is inherently biased relative to the true expected advantage, leading to imbalanced exploration and exploitation In this work, the group-relative advantage estimator is systematically biased relative to the true (expected) advantage, underestimating advantages for hard prompts and overestimating them for easy prompts To address this fundamental issue, the authors propose History-Aware Adaptive Difficulty Weighting (HA-DW), an adaptive reweighting scheme that adjusts advantage estimates based on an evolving difficulty anchor and training dynamics Discovery of Bias The theoretical analysis reveals a systematic bias in group-based RL where the estimator tends to underestimate expected advantages for hard prompts and overestimate them for easy prompts This bias is amplified as the prompt difficulty deviates from 0.5, and it becomes deterministic in extreme regimes, meaning the estimator necessarily underestimates the true advantage for extremely difficult prompts and overestimates it for extremely easy prompts Specifically, Theorem 1 shows that E
Aˆt,i S: At,i if pt > 0.5 Proposed Solution Mechanism The proposed solution is History-Aware Adaptive Difficulty Weighting (HA-DW), which consists of two key components: an evolving difficulty anchor and an adaptive advantage reweighting algorithm The evolving difficulty anchor incorporates cross-batch historical information by propagating the model’s prior through a history buffer, capturing long-term reward trends The second component dynamically adjusts advantage weights using a history-based prompt difficulty and an exponential reweighting factor to compensate for the systematic bias Theoretical Guarantees The theoretical analysis provides strong guarantees regarding the effectiveness of the adjustment strategy, showing that under appropriate scaling factors, HA-DW provably mitigates the bias of group-relative advantage Theorem 3 demonstrates that with an appropriate choice of the scaling parameter λscale, HADW algorithm provably mitigates the bias of group-relative advantage Furthermore, Corollary 3 shows that for extremely difficult prompts (pt < 1/G), the estimator necessarily underestimates the true advantage At,i and for extremely easy prompts (pt > (G − 1)/G), it necessarily overestimates it Empirical Validation Experiments on five mathematical reasoning benchmarks demonstrate consistent performance improvements when HA-DW is integrated with GRPO and its variants across model scales Table 7 shows that when λscale = 1.3 or 1.5, the trained model achieves the best performance across five benchmarks The results underscore that HA-DW compensates for advantage estimation bias via dynamic reweighting to fully leverage these overshadowed critical prompts Ablation and Extensions The authors conducted extensive experiments, including an ablation study on the group size G (Table 3) and an ablation study on the scaling parameter λscale The results suggest that increasing the number of rollouts can, to a certain extent, enhance model performance by providing a more stable baseline The analysis was extended to continuous bounded reward distributions, where the core bias phenomenon remains prevalent across a broader class of bounded reward models Case Study Application In a case study involving two circles, the GRPO solution found that the slope of the line passing through the intersection points is-1, resulting in m + n = 0 The GRPO+HA-DW solution found that the slope is 2/5, resulting in m + n = 7 Final Answer The final answer is 7 The final answer is 7 In the xy-plane, a circle of radius 4 with center on the positive x-axis is tangent to the y-axis at the origin, and a circle with radius 10 with center on the positive y-axis is tangent to the x-axis at the origin.
Tom: What is the slope of the line passing through the two points at which these circles intersect?
Jane: The final answer can be written in the form m n, where m and n are relatively prime positive integers.
Tom: What is m+n?
Paper summary: Tom: So wrapping up this paper, "Your Group-Relative Advantage Is Biased," they’ve laid out that group-relative advantage estimation in RLVR isn't perfectly accurate; it systematically underestimates advantages for hard prompts and overestimates them for easy ones.
Jane: And the key finding is that this bias causes problems with exploration and exploitation during training, especially when dealing with prompts near the five point zero mark.
Lu: They propose History-Aware Adaptive Difficulty Weighting, HA-DW, which uses an evolving difficulty anchor to capture long-term trends and then uses an exponential reweighting factor to dynamically adjust the advantage weights.
Meng: The takeaway for deployment is that this adaptive reweighting scheme provably mitigates that bias if you choose the right scaling parameter, lambda scale, which they showed in their theoretical analysis <ref:2601.08521#pg3>.
Lalam: Empirically, integrating HA-DW with GRPO and its variants consistently improved performance across five mathematical reasoning benchmarks when compared to models without it <ref:2601.08521#pg2>.
Tom: It’s a lot of work to prove that, but the consistency across those benchmarks is what makes this compelling for anyone building RLVR systems today.
Jane: It really shifts the discussion from just getting a policy update rule to ensuring the advantage signal itself is sound and balanced across all prompt difficulties <ref:2601.08521#pg3>.
Conclusion: Tom: So, this paper is all about how those group-relative advantages we use in AI training aren't actually telling us the whole story.
Jane: It looks like they’re calling it "Your Group-Relative Advantage Is Biased" and the authors are looking right at that problem.
Lu: They’re showing that their advantage estimation method has a systematic error, where it underestimates advantages for hard prompts and overestimates them for easy ones.
Meng: So what does that actually mean when you're deploying a model? It means the training signal is skewed against the hardest stuff and too enthusiastic about the easy stuff.
Lalam: Because of this, if you just use those numbers directly, your AI won't explore as well or exploit as effectively during training.
Tom: Exactly. It messes up the whole balance between learning what’s hard and polishing what’s already known.
Jane: The solution they propose is something called History-Aware Adaptive Difficulty Weighting, which basically tries to fix that skew dynamically during training.
Lu: It uses an evolving difficulty anchor that looks at past performance across different batches and then adjusts the advantage weights based on that history and some exponential factor.
Meng: I’m curious about the cost there, though. If you have to track all that historical data for every group adjustment, does it slow down training?
Tom: Well, their analysis shows that with the right scaling parameter, which they call lambda scale, this method actually helps improve performance on those mathematical reasoning benchmarks.
Jane: So what’s the big picture here? It suggests we need to be much more careful about how we score and weigh the AI's learning signals before we trust the final model.
Lu: They prove that if you choose that scaling parameter correctly, this adjustment provably mitigates the bias in group-relative advantage estimation.
Meng: It’s good they proved it works across those benchmarks, but I still need to know how robust this is when you move off those specific math problems and onto real traffic.
Tom: That’s the next thing we need to look at—the generalization of this fix beyond just five mathematical reasoning tasks.
More episodes
- 2603.28595-Optimistic Actor-Critic with Parametric Policies for Linear Markov Decision Processes
- 2603.29765-Training-Free Dynamic Upcycling of Expert Language Models
- 2605.00778-Observable Performance Does Not Fully Reflect Adaptive System Organization: A Multi-Level Analysis of Gait Dynamics Under Occlusal Constraint
- 2605.06906-TraXion: Rethinking Pre-training Frameworks for Mobility and Beyond
- 2605.09165-Sparse Layers are Critical to Scaling Looped Language Models
- 2605.09439-Inverse Design for Conditional Distribution Matching
- 2605.10313-Signature Approach for Contextual Bandits with Nonlinear and Path-dependent Rewards: Regret Bound and Statistical Inference
- 2605.10878-Minimum Neural Weight Norm = Kolmogorov Complexity
- 2605.11617-MIST: Reliable Streaming Decision Trees for Online Class-Incremental Learning via McDiarmid Bound
- 2605.15085-From Data to Action: Accelerating Refinery Optimization with AI