Unveiling Implicit Advantage Symmetry: Why GRPO Struggles with Exploration and Difficulty Adaptation
cs.LG, cs.AI
Submitted: 2026-02-05
Updated: 2026-09-27
Comments: NeurIPS 2026
Code: https://github.com/HKU-HealthAI/A-GRAE
License: http://creativecommons.org/licenses/by/4.0/
The gist: Reinforcement Learning with Verifiable Rewards (RLVR), particularly GRPO, has become the standard for eliciting LLM reasoning.
Terminology
Abstract
Reinforcement Learning with Verifiable Rewards (RLVR), particularly GRPO, has become the standard for eliciting LLM reasoning. However, its efficiency in exploration and difficulty adaptation remains an open challenge. In this work, we identify an implicit advantage symmetry inherent in Group Relative Advantage Estimation (GRAE) as a structural property that provides a new perspective for understanding these bottlenecks. This symmetry induces two critical limitations: (i) at the group level, strict symmetry in weights between correct and incorrect trajectories leaves unsampled action logits unchanged, thereby hindering the exploration of novel correct solution. (ii) at the sample level, the algorithm implicitly prioritizes medium-difficulty samples, remaining agnostic to the non-stationary demands of difficulty focus. Through controlled experiments, we reveal that this symmetric property is sub-optimal, yielding two pivotal insights: (i) asymmetrically down-weighting the advantages of correct trajectories encourages essential exploration but risks instability; (ii) learning efficiency can be boosted by a curriculum-like transition-prioritizing simpler samples initially before gradually shifting to complex ones. Motivated by these findings, we propose Asymmetric GRAE (A-GRAE), which dynamically modulates exploration incentives and sample-difficulty focus. Experiments across seven benchmarks demonstrate that A-GRAE consistently improves GRPO and its variants across both LLMs and MLLMs. Code is available at https://github.com/HKU-HealthAI/A-GRAE
Sources
- GPT-4 Technical Report
- Online Difficulty Filtering for Reasoning Oriented Reinforcement Learning
- XRPO: Pushing the limits of GRPO with Targeted Exploration and Exploitation
- HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scale
- Evaluating Large Language Models Trained on Code
- Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models
- Reasoning with Exploration: An Entropy Perspective
- Boosting the Generalization and Reasoning of Vision Language Models with Curriculum Reinforcement Learning
- Decomposing the Entropy-Performance Exchange: The Missing Keys to Unlocking Effective Reinforcement Learning
- Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs
- EMIT: Enhancing MLLMs for Industrial Anomaly Detection via Difficulty-Aware GRPO
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Rethinking Entropy Interventions in RLVR: An Entropy Change Perspective
- PathVQA: 30000+ Questions for Medical Visual Question Answering
- Measuring Mathematical Problem Solving With the MATH Dataset
- A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibility
- REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization
- Semantic-Space Exploration and Exploitation in RLVR for LLM Reasoning
- OpenAI o1 System Card
- PuzzleCraft: Exploration-Aware Curriculum Learning for Puzzle-Based RLVR in VLMs
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks