One Offset, Two Directions: Baseline Adjustment via Subgroup Offsets
cs.AI
Submitted: 2026-10-31
Updated: 2026-11-13
License: http://creativecommons.org/licenses/by/4.0/
The gist: Group Relative Policy Optimization (GRPO) estimates advantages against the within-prompt mean reward, a baseline shared across the group.
Terminology
Abstract
Group Relative Policy Optimization (GRPO) estimates advantages against the within-prompt mean reward, a baseline shared across the group. In practice, rollouts within a prompt are not always homogeneous, differing in their solution strategies and reasoning trajectories, along which they may group into subgroups. We further find that these heterogeneous subgroups may exhibit distinct reward distributions. The global baseline, however, collapses them onto a single credit axis, where the credit reflects neither a rollout's standing within its subgroup nor the subgroup's standing relative to the rest. We therefore study how to adjust the baseline when the rollouts of a prompt fall into heterogeneous subgroups. We define the subgroup offset as the deviation of a subgroup mean from the global mean, and shift the baseline along it by a single signed coefficient λ. The sign of λ yields two opposing directions: Alignment (λ > 0) adds the offset to the baseline, drawing credit toward a rollout's relative position within its subgroup; Contrast (λ < 0) subtracts it, drawing a rollout's credit toward the relative quality of the subgroup it belongs to. The adjustment acts on the baseline alone, leaving the reward signal and the optimization objective unchanged. In this work, we induce subgroups from strategies, grouping rollouts that share a strategy into the same subgroup. We evaluate both directions on Qwen3-8B-Base and Olmo3-7B-Base across four competition-level math benchmarks, against a λ = 0 anchor that shares the same strategy-conditioned sampling and differs only in the baseline. Against this anchor each direction yields modest but consistent gains in multi-sample coverage, with Contrast the stronger of the two.
Sources
- Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models
- Beyond Trajectory-Level Attribution: Graph-Based Credit Assignment for Agentic Reinforcement Learning
- The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models
- Latent-GRPO: Group Relative Policy Optimization for Latent Reasoning
- Agentic Reinforced Policy Optimization
- Plan Then Action:High-Level Planning Guidance Reinforcement Learning for LLM Reasoning
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigation
- MC-GRPO: Median-Centered Group Relative Policy Optimization for Small-Rollout Reinforcement Learning
- Tulu 3: Pushing Frontiers in Open Language Model Post-Training
- No Prompt Left Behind: Exploiting Zero-Variance Prompts in LLM Reinforcement Learning via Entropy-Guided Advantage Shaping
- Understanding R1-Zero-Like Training: A Critical Perspective
- From Absolute to Relative: Rethinking Reward Shaping in Group-Based Reinforcement Learning
- Olmo 3
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoning
- Exploiting Tree Structure for Credit Assignment in RL Training of LLMs
- Implicit Hierarchical GRPO: Decoupling Tool Invocation from Execution for Tool-Integrated Mathematical Reasoning
- RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning
- Negative Advantages Is a Double-Edged Sword: Calibrating advantages in GRPO for Search Agents
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents