Wrong the Same Way: Cross-Prompt Sharing Both Suppresses and Amplifies Reward Hacking in RLVR
cs.LG, cs.AI
Submitted: 2026-11-09
Updated: 2026-11-09
Code: https://github.com/Rohan-Pandey1729/RLVR-Reward-Hacking
Terminology
Sources
- Gradient Regularization Mitigates Reward Hacking in Reinforcement Learning from Human Feedback and Verifiable Rewards
- Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers
- Coherent Gradients: An Approach to Understanding Generalization in Gradient Descent-based Optimization
- Binary Rewards and Reinforcement Learning: Fundamental Challenges
- Delay, Plateau, or Collapse: Evaluating the Impact of Systematic Verification Error on RLVR
- Differential Smoothing Mitigates Sharpening and Improves LLM Reasoning
- LLMs Gaming Verifiers: RLVR can Lead to Reward Hacking
- From Accuracy to Robustness: A Study of Rule- and Model-based Verifiers in Mathematical Reasoning
- Countdown-Code: A Testbed for Studying The Emergence and Generalization of Reward Hacking in RLVR
- Let's Verify Step by Step
- Understanding R1-Zero-Like Training: A Critical Perspective
- Uniform-Correct Policy Optimization: Breaking RLVR's Indifference to Diversity
- Natural Emergent Misalignment from Reward Hacking in Production RL
- Towards Understanding Specification Gaming in Reasoning Models
- An Imperfect Verifier is Good Enough: Learning with Noisy Rewards
- Rate or Fate? RLV$^\varepsilon$R: Reinforcement Learning with Verifiable Noisy Rewards
- Learning Dynamics of LLM Finetuning
- Spurious Rewards: Rethinking Training Signals in RLVR
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Learning Dynamics in RL Post-Training for Language Models
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis