KV-PRM: Efficient Process Reward Modeling via KV-Cache Transfer for Multi-Agent Test-Time Scaling
cs.AI
Submitted: 2026-07-10
Updated: 2026-11-04
Comments: Accepted at NeurIPS 2026. 27 pages, including appendices
Code: https://github.com/RLHFlow/RLHF-Reward-Modeling
License: http://creativecommons.org/licenses/by/4.0/
The gist: Process Reward Models (PRMs) have been proven to be highly effective in guiding test-time scaling (TTS) methods, which significantly boost the capabilities of LLM-based multi-agent systems.
Terminology
Abstract
Process Reward Models (PRMs) have been proven to be highly effective in guiding test-time scaling (TTS) methods, which significantly boost the capabilities of LLM-based multi-agent systems. However, existing PRMs are text-based: they re-encode the entire trajectory text from scratch. In long multi-agent rollouts, the scoring cost, growing quadratically with respect to sequence length L, creates a severe computational bottleneck, severely limiting PRMs' application in long-context scenarios. To resolve this, we introduce KV-PRM, a highly efficient process reward model that eliminates the heavy text re-encoding by directly reading the KV cache produced naturally during the LLM's generation phase. By processing a single "verify token" against the pre-existing KV cache, KV-PRM reduces the scoring cost from O(L 2) to O(L). We formally prove that the KV cache contains strictly greater information capacity than text, and is more efficient for downstream reward modeling. Empirically, across the MATH, GSM8K, and AIME benchmarks, KV-PRM matches or strictly outperforms text-PRMs under various TTS methods such as Beam Search, MCTS, and Weighted Voting, with up to a 5,000x reduction in scoring FLOPs, a 37x reduction in latency, and a 34x reduction in per-sequence memory footprint compared to text-based PRMs.
Sources
- Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Let's Verify Step by Step
- Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations
- A Survey of Process Reward Models: From Outcome Signals to Process Supervisions for Large Language Models
- LoRA: Low-Rank Adaptation of Large Language Models
- MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework
- MASPRM: Multi-Agent System Process Reward Model
- Self-Consistency Improves Chain of Thought Reasoning in Language Models
- Latent Collaboration in Multi-Agent Systems
- The Spectral Geometry of Thought: Phase Transitions, Instruction Reversal, Token-Level Dynamics, and Perfect Correctness Prediction in How Transformers Reason
- Training Verifiers to Solve Math Word Problems
- Qwen3 Technical Report
- Process Reward Models That Think
- GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoning
- Improve Mathematical Reasoning in Language Models by Automated Process Supervision
- Generative Verifiers: Reward Modeling as Next-Token Prediction
- Improving Factuality and Reasoning in Language Models through Multiagent Debate
- A Survey on Large Language Model Acceleration based on KV Cache Management
- Efficient Streaming Language Models with Attention Sinks
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents