Fast-Slow Thinking RM: Efficient Integration of Scalar and Generative Reward Models
cs.CL, cs.LG
Submitted: 2026-03-02
Updated: 2026-09-26
Comments: Accepted to the EMNLP 2026 Main Conference
License: http://creativecommons.org/licenses/by/4.0/
The gist: Reward models (RMs) are critical for aligning Large Language Models via Reinforcement Learning from Human Feedback (RLHF).
Terminology
Abstract
Reward models (RMs) are critical for aligning Large Language Models via Reinforcement Learning from Human Feedback (RLHF). While Generative Reward Models (GRMs) achieve superior accuracy through chain-of-thought (CoT) reasoning, they incur substantial computational costs. Conversely, Scalar Reward Models (SRMs) offer efficiency but suffer from limited performance and adaptability in complex scenarios. We introduce Fast-Slow Thinking Reward Models (F/S-RM), a hybrid RM architecture inspired by Dual Process Theory. It trains a single model to integrate two distinct reward paradigms: scalar-style first-token pairwise judgment (fast thinking) and CoT-based judgment (slow thinking), regulated by a dual-confidence activation mechanism that determines when to activate slow thinking. Under hybrid inference, F/S-RM achieves state-of-the-art accuracy with an average score of 84.3 across benchmarks, while reducing token consumption by 22.5%.
Sources
- BLEUBERI: BLEU is a surprisingly effective reward for instruction following
- RM-R1: Reward Modeling as Reasoning
- Deep Think with Confidence
- Reward Reasoning Model
- Think Twice: Branch-and-Rethink Reasoning Reward Model
- Scalable Best-of-N Selection for Large Language Models via Self-Certainty
- LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods
- Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs
- Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy
- RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style
- Inference-Time Scaling for Generalist Reward Modeling
- Generative Reward Models
- HybridFlow: A Flexible and Efficient RLHF Framework
- JudgeBench: A Benchmark for Evaluating LLM-based Judges
- Self-rationalization improves LLM as a fine-grained judge
- PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization
- J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning
- reWordBench: Benchmarking and Improving the Robustness of Reward Models with Transformed Inputs
- Don't Overthink It: A Survey of Efficient R1-style Large Reasoning Models
- Generative Verifiers: Reward Modeling as Next-Token Prediction
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering