ReuseGR: Offline--Online Decoupled Generative Retrieval via Adaptive Prefix Reuse
cs.IR
Submitted: 2026-11-03
Updated: 2026-11-03
License: http://creativecommons.org/licenses/by/4.0/
The gist: Larger generative retrieval models can improve retrieval quality, but their decoding cost makes low-latency serving difficult.
Terminology
Abstract
Larger generative retrieval models can improve retrieval quality, but their decoding cost makes low-latency serving difficult. When different users submit the same query, generating complete Semantic IDs (SIDs) for every request repeats computation, while their interaction histories call for different results. The challenge is to reuse large-model computation across requests while keeping retrieval personalized. We propose ReuseGR, which separates query-level candidate planning from request-level personalized completion. A 7B offline planner generates SID prefix sets shared across requests with the same query. A 0.5B online model uses each request's history to select prefixes and generate the remaining tokens. ReuseCut chooses a cache depth for each query based on historical coverage, balancing computation reuse and candidate coverage. To strengthen online completion, we transfer 7B suffix-scoring knowledge to 0.5B through path-level knowledge distillation (Path KD). BudgetCut then selects scored branches to improve final top-100 recall under a fixed branch budget. Its selection policy is initialized by on-policy distillation (OPD) and refined through group-relative policy optimization (GRPO). On an industrial search-advertising dataset with approximately 2M query--item records and 3.38M catalog items, ReuseGR improves query-level HR@100 by 3.83 percentage points and reduces mean online latency by 38.9% relative to an all-online 0.5B baseline when prefixes are prepared. It achieves HR@100 comparable to an all-online 7B baseline with 57.7% lower online latency. A 14-day user-level online A/B test against the 0.5B baseline shows relative gains of 0.93% in click-through rate and 1.05% in revenue per thousand requests.
Sources
- On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes
- OneSearch: A Preliminary Exploration of the Unified End-to-End Generative Framework for E-commerce Search
- OneSearch-V2: The Latent Reasoning Enhanced Self-distillation Generative Search Framework
- OneRec: Unifying Retrieve and Rank with Generative Recommender and Iterative Preference Alignment
- MLPs are Efficient Distilled Generative Recommenders
- Distilling the Knowledge in a Neural Network
- Efficient Inference for Large Language Model-based Generative Recommendation
- Qwen2.5 Technical Report
- Proximal Policy Optimization Algorithms
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Vectorizing the Trie: Efficient Constrained Decoding for LLM-based Generative Retrieval on Accelerators
- Differentiable Geometric Indexing for End-to-End Generative Retrieval
- Learning to Reflect and Correct: Towards Better Decoding Trajectories for Large-Scale Generative Recommendation
- SmartGR: Hierarchy and Beam-Aware Knowledge Distillation for Generative Recommendation
- RcLLM: Accelerating Generative Recommendation via Beyond-Prefix KV Caching
Related papers
- UniRank: Benchmarking Ranking Models for Unified Sequential Modeling and Feature Interaction
- Revisiting Text Ranking in Deep Research
- GateSID: One Gate to Balance Semantic and Collaborative Signals across the Item Lifecycle
- SAM-D2Q: Aligning Multimodal Doc2Query with Search Demand and Conversion for E-commerce
- GINGER: Spicing Up Graph-Based Vector Search
- Subtraction Gets You More: Gap-Aware Retrieval for Multimodal Multi-Hop QA