Learning from What You Retrieve: Online RL Fine-Tuning for Semantic Retrieval
cs.IR, cs.AI
Submitted: 2026-08-31
Updated: 2026-08-31
License: http://creativecommons.org/licenses/by/4.0/
The gist: In large-scale e-commerce retrieval, dual-encoder retrievers are op- timized for contrastive similarity, whereas downstream rerankers capture finer-grained relevance preferences; this objective mis-
Terminology
Abstract
In large-scale e-commerce retrieval, dual-encoder retrievers are op- timized for contrastive similarity, whereas downstream rerankers capture finer-grained relevance preferences; this objective mis- match limits end-to-end retrieval quality. Reinforcement Learning offers a way to use reward-model feedback for retriever adaptation, but we observe that standard policy-gradient updates can degrade embedding geometry, especially when the document index must remain frozen due to industrial constraints. To address this, we propose PAO (Positive-Advantage-Only), a selective RL optimization method. Our analysis reveals that in- discriminate penalization of negative samples (pushing away) in a frozen high-dimensional space disrupts pre-trained semantic man- ifolds. PAO selectively applies gradient updates only to retrieved items with positive advantages, effectively pulling query embed- dings toward high-reward regions while preserving global topo- logical stability. Experiments on both a massive industrial dataset and public benchmarks demonstrate that PAO significantly outper- forms standard RL and distillation baselines.
Sources
- MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
- M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation
- Towards General Text Embeddings with Multi-stage Contrastive Learning
- Retrieval-GRPO: A Multi-Objective Reinforcement Learning Framework for Dense Retrieval in Taobao Search
- Passage Re-ranking with BERT
- Qwen3 Technical Report
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Representation Learning with Contrastive Predictive Coding
- C-Pack: Packed Resources For General Chinese Embeddings
- Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
- mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval
Related papers
- The Price of Isolation: Estimating the Ecosystem Cost of Symmetric Two-Sided A/B Testing
- SCAR: Semantic Continuity-Aware Retrieval for Efficient Context Expansion in RAG
- MixLoRA-DSI: Dynamically Expandable Mixture-of-LoRA Experts for Rehearsal-Free Generative Retrieval over Dynamic Corpora
- QProtFuse: Query-Guided Layer Fusion for Retrieval-Augmented Protein Function Prediction
- DualView: Adaptive Local-Global Fusion for Multi-Hop Document Reranking
- How Do LLM-Generated Texts Impact Term-Based Retrieval Models?