Verifier Errors in RLVR: Reward Hacking, Limits of Feedback, and Selective Control

arXiv:2609.35677 · cs.AI · Submitted 2026-09-28 · Read on arXiv

cs.AI

Submitted: 2026-09-28

Updated: 2026-09-28

Code: https://github.com/cmoyacal/verifier-errors

Terminology

• Reward Hacking — 1× in this paper · explained in 8 episodes
• Verifier — 1× in this paper · explained in 2 episodes

Sources

• Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation
• IR$^3$: Contrastive Inverse Reinforcement Learning for Interpretable Detection and Mitigation of Reward Hacking
• Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers
• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
• Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models
• Interpreting Reinforcement Learning Agents with Susceptibilities
• Explaining and Preventing Alignment Collapse in Iterative RLHF
• LLMs Gaming Verifiers: RLVR can Lead to Reward Hacking
• Countdown-Code: A Testbed for Studying The Emergence and Generalization of Reward Hacking in RLVR
• Natural Emergent Misalignment from Reward Hacking in Production RL
• Spontaneous Reward Hacking in Iterative Self-Refinement
• Patterning: The Dual of Interpretability
• Reproducing, Analyzing, and Detecting Reward Hacking in Rubric-Based Reinforcement Learning
• ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases

Related papers

• MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
• Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
• Agents' Last Exam
• The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
• MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
• Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems