RH-Detect: A Unified Benchmark for Reward Hacking Detection

The gist The researchers present RH-Detect, a common benchmark that harmonizes reward-hacking-relevant subsets from eleven public datasets using consistent metadata, task-specific tracks, and audited source-held-out splits.

How it works

RHDetect is a unified benchmark that combines reward hacking relevant subsets from eleven public datasets into a common schema to facilitate comparison On 5,021 open-ended evaluation units, each comprising a task prompt and a free-form model continuation, including multi-turn tooluse trajectories, the researchers evaluate six off-the-shelf language models from five families as reward hacking detectors without additional training. The best model achieves a pooled AUROC of 0.962, with accuracy above 93%.

Taxonomy and Data Construction

The benchmark incorporates a six behavior category taxonomy derived from surveying selected source datasets to harmonize different labeling schemes These categories include Output style gaming, Evaluation gaming, Reward tampering, Environment exploitation, Deception, and Sycophancy. The construction process involves selecting reward-hacking relevant subsets from eleven public sources and mapping them into a common schema while preserving inherited label semantics and source provenance. Metadata preserved for each row includes construction method, verification method, a free-text hacking description when provided by the source, the response model that generated the continuation, and an agentic flag indicating multi-turn trajectories.

Detector Evaluation Results

The zero-shot evaluation involves asking a fixed detector prompt to classify continuations as reward hacking or clean and return a JSON with reasoning and a probability from 0 to 100. The top three models in the pooled ranking are Qwen 3.5 Flash (AUROC 0.962), Gemini 3.1 Flash Lite (0.960), and Grok 4.1 Fast (AUROC 0.960 on reduced coverage). However, the four strongest models exhibit a gap when evaluated on MALT and TRACE, where their accuracy is 10.7–15.9 percentage points lower than on other sources at a common decision threshold.

Input Representation and Training Analysis

The study investigates how detector performance varies with input representation, comparing conditions like E1 (plain-text), E2 (stripping extended-reasoning spans), E3 (retaining only reasoning spans), and E4/E4b (extract-and-evaluate design). For the Qwen3.5-4B judge, removing input reasoning raises AUROC from 0.779 to 0.849, whereas retaining only the reasoning lowers it to 0.655. Single-token supervised fine-tuning improves average AUROC per-source by 0.036 but shows uneven transfer, with a notable drop on School of Reward Hacks (SoRH).

Post-Training Strategies

The paper evaluates different posttraining procedures to test robustness, including single-token SFT under six strict leave-one-source-out rotations and a targeted GRPO follow-up for the SoRH failure case. The GRPO study on SoRH excluded from training yields a small positive gain of approximately +0.018 on the held-out source, showing that an alternative post-training objective can avoid the observed regression.

Conclusion

Reliable reward-hacking monitoring requires sourcestratified evaluation and complete specification of the detector input, score readout, and training procedure. The best models are less accurate on MALT and TRACE than on other sources, highlighting multi-turn tool-use monitoring as an area needing more engineering investment. Reliable detection is a prerequisite for downstream reward hacking interventions

--- Page 1 ---

RH-Detect: A Unified Benchmark for Reward Hacking Detection Junwei Quan1∗, Evgenii Opryshko1,2,4∗, Rohan Subramani1,2,3,4, Igor Gilitschenski1,2 1University of Toronto 2Vector Institute 3Aether Research 4Trajectory Labs Abstract Reward hacking where a model exploits an evaluation signal without completing the intended task threatens the reliability of deployed language model systems. Existing datasets use different labels, response formats, and metadata conventions, making detector results difficult to compare. We present RHDetect, a benchmark that combines reward-hacking-relevant subsets from eleven public datasets, comprising 92,761 rows and six behavior categories, into a common schema. On 5,021 open-ended evaluation units, each comprising a task prompt and a free-form model continuation, including multi-turn tooluse trajectories, we evaluate six off-the-shelf language models from five families as reward hacking detectors without additional training. The best model achieves a pooled AUROC of 0.962, with accuracy above 93%. For the four strongest models, however, accuracy on the two multi-turn tool-use datasets, MALT and TRACE, is 10.7–15.9 percentage points lower than on the other sources at a common decision threshold, highlighting a key gap for deployment-time monitoring. We find that different input formats have different effects across models. Removing thinking raises Qwen3.5-4B AUROC from 0.779 to 0.849, but lowers Qwen Flash from 0.977 to 0.950. We also evaluate the benchmark as a training dataset for detectors. Holding out each source in turn, single-token SFT improves average AUROC on five of six held-out sources. A GRPO follow-up on that failure case yields a slight improvement in detection performance. Our results show that a single pooled score can conceal variation across data sources, detector inputs, and training procedures. 1 Introduction Language models can exploit imperfections in training or evaluation signals, achieving high measured reward without completing the intended task. Such behavior ranges from sycophancy and test-case exploitation to reward tampering and concealment under monitoring (Gao, Schulman, and Hilton 2023; Sharma et al. 2024; Denison et al. 2024; Baker et al. 2025). Recent work further shows that learning to reward hack in production reinforcement-learning environments can induce broader emergent misalignment, including alignment faking, cooperation with malicious actors, and attempted sabotage (MacDiarmid et al. 2025). We use reward hacking broadly to mean behavior that satisfies an evaluation signal without fulfilling the intended task, including behavior observed outside the training process, which is also referred to as “evaluation gaming” (Nishimura-Gasparian et al. 2024; Denison et al. 2024). Reward hacking is therefore not only a performance or evaluation problem: it may also undermine deployment-time oversight as models are entrusted with higher-stakes tasks. Several research groups have independently released datasets related to reward hacking (Taylor et al. 2025; Nishimura-Gasparian et al. 2024; Deshpande, Kannappan, and Qian 2026; Parikh and Wijk 2025; Perez et al. 2013; Jozdien 18) Each source, however, uses its own labeling scheme, response format, metadata conventions, and evaluation procedure, and several contain traces irrelevant to reward hacking. This fragmentation mirrors a broader problem in behavioral testing: aggregate scores are difficult to interpret when task definitions and evaluation pipelines are not standardized (Ribeiro et al. 2020). A practitioner who wants to evaluate one monitoring pipeline across a broad range of hacking behaviors must currently build source-specific ingestion, filtering, and interpretation pipelines, and a strong result on one dataset provides limited evidence that the same detector will transfer to an unseen source. We address this gap with RH-Detect, a common framework that harmonizes reward-hacking-relevant subsets from eleven public datasets. The benchmark provides a shared schema, a six-category behavior taxonomy, provenance metadata, task-specific tracks, and source-held-out splits audited by exact prompt hash. We use it to answer three research questions: RQ1 How effectively can language models detect reward hacking without additional training, and how does performance vary across model families? RQ2 How does the monitor performance differ across data sources and reward hacking categories? RQ3 How sensitive is detector performance to transcript representation, and how well do supervised and RL posttraining methods transfer to held-out data sources and arXiv:2610.10947v1 [cs.LG] 7 Oct 2026

--- Page 2 ---

task tracks? RQ1 tests whether existing language models can serve as low-cost monitors without task-specific training RQ2 identifies where strong aggregate performance hides source- or behavior-specific weaknesses RQ3 tests whether changes to detector input or training procedure improve cross-source robustness rather than only aggregate performance. Reliable detection is a prerequisite for downstream reward hacking interventions (Mallen 2025)

--- Page 3 ---

Our contributions are as follows. 1 A unified benchmark that selects and maps rewardhacking-relevant subsets from eleven public datasets, comprising 92,761 rows, into a common schema while preserving inherited label semantics, source provenance, and task-specific formats.

1. Identify reward hacking through a unified benchmark that combines reward-hacking-relevant subsets from eleven public datasets, comprising 92,761 rows and six behavior categories into a common schema. This allows for consistent comparison across different data sources and behavioral types.

2. Develop detector input representations by testing transcript preparation conditions such as E1 passes the plain-text transcript, E2 strips extended-reasoning (“thinking”) spans from responses, and E4 first extracts task information and suspicion-relevant evidence verbatim. This enables researchers to understand how different transcript formats affect detection performance.

3. Implement posttraining strategies by evaluating methods like single-token SFT and a GRPO follow-up to see if they improve robustness, noting that single-token SFT improves average AUROC on five of six held-out sources.

4. Create source-stratified monitoring pipelines by recognizing that performance is not uniform, as the paper shows, for instance, the four strongest complete-coverage judges are 10.7–15.9 percentage points less accurate on MALT/TRACE than on the other sources.

5. Design detection protocols that incorporate evidence extraction to preserve strong judge performance; specifically, E4b keeps the Qwen Flash judge fixed but uses GPT-5 Nano as the extractor, which can preserve Qwen Flash near its raw-transcript result (0.958) but lowers GPT-5 Nano to 0.788.

Reward hacking, where a model exploits an evaluation signal without completing the intended task, threatens the reliability of deployed language model systems. Existing datasets use different labels, response formats, and metadata conventions, making detector results difficult to compare. We present RH-Detect, a benchmark that combines reward-hacking-relevant subsets from eleven public datasets, comprising 92,761 rows and six behavior categories, into a common schema. On 5,021 open-ended evaluation units, each comprising a task prompt and a free-form model continuation, including multi-turn tool-use trajectories, we evaluate six off-the-shelf language models from five families as reward hacking detectors without additional training. The best model achieves a pooled AUROC of 0.962, with accuracy above 93%. For the four strongest models, however, accuracy on the two multi-turn tool-use datasets, MALT and TRACE, is 10.7-15.9 percentage points lower than on the other sources at a common decision threshold, highlighting a key gap for deployment-time monitoring. We find that different input formats have different effects across models. Removing thinking raises Qwen3.5-4B AUROC from 0.779 to 0.849, but lowers Qwen Flash from 0.977 to 0.950. We also evaluate the benchmark as a training dataset for detectors. Holding out each source in turn, single-token SFT improves average AUROC on five of six held-out sources. A GRPO follow-up on that failure case yields a slight improvement in detection performance. Our results show that a single pooled score can conceal variation across data sources, detector inputs, and training procedures.