STAR-GRPO: Canonical Anchoring and Reliability-First Advantages against Representation-Dependent Reward Hacking

arXiv:2609.36900 · cs.AI · Submitted 2026-09-29 · Read on arXiv

cs.AI

Submitted: 2026-09-29

Updated: 2026-09-29

Terminology

• GRPO — 1× in this paper · explained in 6 episodes
• Reward Hacking — 1× in this paper · explained in 8 episodes

Sources

• HealthBench: Evaluating Large Language Models Towards Improved Human Health
• Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
• Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
• Conformal Feedback Alignment: Quantifying Answer-Level Reliability for Robust LLM Alignment
• Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators
• Scaling Laws for Reward Model Overoptimization
• Understanding R1-Zero-Like Training: A Critical Perspective
• Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy
• RRM: Robust Reward Model Training Mitigates Reward Hacking
• Training language models to follow instructions with human feedback
• Proximal Policy Optimization Algorithms
• DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
• Do we need to estimate the variance in robust mean estimation?
• From Rebound to Remedy: Understanding and Mitigating Reward Hacking via Representation Engineering
• DAPO: An Open-Source LLM Reinforcement Learning System at Scale
• Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensembles
• NoveltyBench: Evaluating Language Models for Humanlike Diversity
• Beyond Semantic Manipulation: Token-Space Attacks on Reward Models
• WildChat: 1M ChatGPT Interaction Logs in the Wild
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

Related papers

• MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
• Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
• Agents' Last Exam
• The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
• MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
• Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems