Neglected Free Lunch from Post-training: Progress Advantage for LLM Agents
cs.LG, cs.AI
Submitted: 2026-06-24
Updated: 2026-11-02
Comments: NeurIPS 2026; Best Paper Awards at ICML 2026 RLxF Workshop
License: http://creativecommons.org/licenses/by/4.0/
The gist: Process reward models enable fine-grained, step-level evaluation of LLMs, yet building them for agentic settings remains prohibitively difficult: long-horizon interactions, irreversible actions…
Terminology
Abstract
Process reward models enable fine-grained, step-level evaluation of LLMs, yet building them for agentic settings remains prohibitively difficult: long-horizon interactions, irreversible actions, and stochastic environment feedback make both human annotation and Monte Carlo estimation infeasible at scale. In this work, we show that reinforcement learning (RL) post-training already provides the ingredients for effective step-level scoring, eliminating the need for dedicated reward model training altogether. Concretely, we derive an implicit advantage under a general stochastic Markov decision process, which we term progress advantage -- log-probability ratio between the RL-trained policy and its reference policy exactly recovers the optimal advantage function. This formulation makes the resulting signal annotation-free, domain-agnostic, and available as a byproduct of the standard RL post-training pipeline. We validate the effectiveness of the progress advantage across three different applications: test-time scaling, uncertainty quantification, and failure attribution on five benchmarks and four model families. Across all settings, it consistently outperforms confidence-based baselines and, despite requiring no task-specific training, surpasses dedicated trained reward models. We complement these results with deeper analyses on characteristics of progress advantage, offering practical guidance for adoption in real-world agentic systems.
Sources
- Training Verifiers to Solve Math Word Problems
- Solving math word problems with process- and outcome-based feedback
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Improve Mathematical Reasoning in Language Models by Automated Process Supervision
- Spurious Rewards: Rethinking Training Signals in RLVR
- Process Reward Models for LLM Agents: Practical Framework and Directions
- $\tau^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment
- Qwen3.5-Omni Technical Report
- Qwen3 Technical Report
- Olmo 3
- Proximal Policy Optimization Algorithms
- On a few pitfalls in KL divergence gradient estimation for RL
- Rethinking the Trust Region in LLM Reinforcement Learning
- WildReward: Learning Reward Models from In-the-Wild Human Interactions
- Process Reward Models That Think
- Uncertainty Quantification in LLM Agents: Foundations, Emerging Challenges, and Opportunities
- Oops, Wait: Discourse Tokens Matter in Reasoning Model
- Efficient Process Reward Modeling via Contrastive Mutual Information
- Process Reinforcement through Implicit Rewards
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis