UR squared: Unify RAG and Reasoning through Reinforcement Learning
cs.CL, cs.AI
Submitted: 2025-08-08
Updated: 2026-09-22
License: http://creativecommons.org/licenses/by/4.0/
The gist: Large Language Models (LLMs) have shown strong capabilities through two complementary paradigms: Retrieval-Augmented Generation (RAG) for knowledge grounding and Reinforcement Learning from
Terminology
Abstract
Large Language Models (LLMs) have shown strong capabilities through two complementary paradigms: Retrieval-Augmented Generation (RAG) for knowledge grounding and Reinforcement Learning from Verifiable Rewards (RLVR) for complex reasoning. However, existing attempts to unify these paradigms remain narrow in scope, typically limited to open-domain QA with fixed retrieval settings, which constrains generalization to broader domains. To address this limitation, we propose UR squared (Unified RAG and Reasoning)), a general reinforcement learning framework that dynamically coordinates retrieval and reasoning. UR squared introduces two key designs: a difficulty-aware curriculum that selectively invokes retrieval only for challenging instances, and a hybrid knowledge access strategy that combines domain-specific offline corpora with on-the-fly LLM-generated summaries. Together, these components mitigate the imbalance between retrieval and reasoning and improve robustness to noisy information. Experiments on open-domain QA, MMLU-Pro, medical, and mathematical reasoning tasks show that UR squared, built on Qwen-2.5-3/7B and LLaMA-3.1-8B, consistently outperforms existing RAG and RL baselines, and achieves performance comparable to GPT-4o-mini and GPT-4.1-mini on several benchmarks. Code, models, and data are available at https://github.com/Tsinghua-dhy/UR2.
Sources
- An Empirical Study on Eliciting and Improving R1-like Reasoning Models
- Training Verifiers to Solve Math Word Problems
- From Local to Global: A Graph RAG Approach to Query-Focused Summarization
- Beyond Ten Turns: Unlocking Long-Horizon Agentic Search with Large-Scale Asynchronous RL
- Vortex counting and velocimetry for slitted superconducting thin strips
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Synthetic Data RL: Task Definition Is All You Need
- Measuring Mathematical Problem Solving With the MATH Dataset
- REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization
- OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework
- Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model
- RAG-RL: Advancing Retrieval-Augmented Generation via RL and Curriculum Learning
- Atlas: Few-shot Learning with Retrieval Augmented Language Models
- Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning
- Understanding and Diagnosing Deep Reinforcement Learning
- WebSailor-V2: Bridging the Chasm to Proprietary Agents via Synthetic Data and Scalable Reinforcement Learning
- Search-o1: Agentic Search-Enhanced Large Reasoning Models
- General-Reasoner: Advancing LLM Reasoning Across All Domains
- Deterministic Implementations for Reproducibility in Deep Reinforcement Learning
- Graph Retrieval-Augmented Generation: A Survey
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering