PRO-Step: Step-level Process Reward Optimization for Retrieval-Augmented Generation

arXiv:2609.01658 · cs.CL, cs.AI · Submitted 2026-08-31 · Read on arXiv

cs.CL, cs.AI

Submitted: 2026-08-31

Updated: 2026-08-31

Comments: 22 pages, 7 figures, 23 tables. Accepted to EMNLP 2026

Code: https://github.com/keemminnke/PRO-Step

License: http://creativecommons.org/licenses/by/4.0/

The gist: Retrieval-Augmented Generation enhances Large Language Models by grounding responses in external knowledge, but multi-hop reasoning remains vulnerable to error propagation, where early retrieval

Terminology

Abstract

Retrieval-Augmented Generation enhances Large Language Models by grounding responses in external knowledge, but multi-hop reasoning remains vulnerable to error propagation, where early retrieval failures confound subsequent steps. Standard outcome-based optimization only rewards the final answer, leaving intermediate retrieval and reasoning errors undetected. While existing process-based methods introduce step-level signals, they still score each step against the final answer, rewarding spurious successes where flawed retrieval coincidentally produces the correct answer. Step-level supervision in RAG requires evaluating both logical validity and evidential grounding at each step. We introduce PRO-STEP: we train a generative PRM that evaluates both dimensions, employ PRM-guided value tree search to construct preference pairs contrasting valid steps against flawed ones, and optimize the policy via step-level Direct Preference Optimization. Experiments on single and multi-hop QA datasets demonstrate that PRO-STEP achieves the best average EM and F1 across five benchmarks. Code, models, and training data are publicly available at https://github.com/keemminnke/PRO-Step.

Sources

Related papers