Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Use
cs.LG, cs.AI
Submitted: 2026-05-03
Updated: 2026-05-03
Comments: 16 pages, 2 figures. Accepted to ICML 2026
Journal ref: Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:120372-120389, 2026
License: http://creativecommons.org/licenses/by/4.0/
The gist: Reinforcement learning (RL) trained language model agents with tool access are increasingly deployed in coding assistants, research tools, and autonomous systems.
Terminology
Abstract
Reinforcement learning (RL) trained language model agents with tool access are increasingly deployed in coding assistants, research tools, and autonomous systems. We introduce the Reward Hacking Benchmark (RHB), a suite of multi-step tasks requiring sequential tool operations with naturalistic shortcut opportunities such as skipping verification steps, inferring answers from task-adjacent metadata, or tampering with evaluation-relevant functions. RHB supports independent and chained task regimes, where chain length acts as a proxy for longer-horizon agent behavior. We evaluate 13 frontier models from OpenAI, Anthropic, Google, and DeepSeek. Exploit rates range from 0% (Claude Sonnet 4.5) to 13.9% (DeepSeek-R1-Zero), varying sharply by post-training style. A controlled sibling comparison (DeepSeek-V3 vs. DeepSeek-R1-Zero) shows RL post-training is associated with substantially higher reward hacking (0.6% vs. 13.9%), with consistent gaps across all four task families. We identify six exploit categories and find that 72% of reward hacking episodes include explicit chain-of-thought rationale, suggesting models often frame exploits as legitimate problem-solving. Simple environmental hardening reduces exploit rates by 5.7 percentage points (87.7% relative) without degrading task success. Models with near-zero exploit rates on standard tasks show elevated rates on harder variants, suggesting that production-aligned post-training appears to suppress reward hacking only below a complexity threshold where honest solutions remain tractable.
Sources
- Benchmark Inflation: Revealing LLM Performance Gaps Using Retro-Holdouts
- INTELLECT-2: A Reasoning Model Trained Through Globally Decentralized Reinforcement Learning
- SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
- Measuring AI Ability to Complete Long Software Tasks
- Natural Emergent Misalignment from Reward Hacking in Production RL
- Large Language Models Often Know When They Are Being Evaluated
- Humanity's Last Exam
- School of Reward Hacks: Hacking harmless tasks generalizes to misaligned behavior in LLMs
- ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis