Autoregressive Retriever: Improving Query Understanding from Item Feedback for Universal Multimodal Retrieval
cs.IR, cs.CV
Submitted: 2026-10-08
Updated: 2026-10-08
Comments: Under Review
License: http://creativecommons.org/licenses/by/4.0/
The gist: Universal multimodal retrieval typically encodes a query once and ranks independently indexed items by embedding similarity.
Terminology
Abstract
Universal multimodal retrieval typically encodes a query once and ranks independently indexed items by embedding similarity. This design supports efficient search, but leaves the query representation unchanged even when retrieved items could help clarify the information need. We introduce the AutoRegressive Retriever (ARR), a multimodal retrieval model that learns both to select informative items and to use their content to refine subsequent retrieval. ARR alternates between retrieving an item and updating the query embedding, then uses the final embedding to rank the collection. Supervised fine-tuning teaches the encoder to use feedback through stepwise contrastive supervision. Reinforcement learning treats feedback items as actions and optimizes their selection using the final reciprocal rank of a relevant item. A query-side adapter enables this optimization against a fixed item index. ARR demonstrates strong retrieval performance on both in-domain and zero-shot benchmarks, outperforming the compared baselines on average. Further analyses show that feedback improves retrieval at inference time and that training with feedback also improves the initial query embedding, before any item is observed.
Sources
- ADORE: Iterative Query Expansion with Retrieval-Grounded Relevance Feedback
- TRACE: Task-Adaptive Reasoning and Representation Learning for Universal Multimodal Retrieval
- E5-V: Universal Embeddings with Multimodal Large Language Models
- Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Ranking
- ELVA: Exploring Ranking-Driven Universal Multimodal Retrieval
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- QueryAgent-R1: Bridging Query Generation and Product Retrieval for E-Commerce Query Recommendation
- Bottleneck Tokens for Unified Multimodal Retrieval
- LaME: Learning to Think in Latent Space for Multimodal Embedding via Information Bottleneck
- Think When Needed: Adaptive Reasoning-Driven Multimodal Embeddings with a Dual-LoRA Architecture
- OneRetrieval: Unifying Multi-Branch E-commerce Retrieval with an Editable Generative Model
- GRAPE: Let GRPO Supervise Query Rewriting by Ranking for Retrieval
Related papers
- The Price of Isolation: Estimating the Ecosystem Cost of Symmetric Two-Sided A/B Testing
- SCAR: Semantic Continuity-Aware Retrieval for Efficient Context Expansion in RAG
- MixLoRA-DSI: Dynamically Expandable Mixture-of-LoRA Experts for Rehearsal-Free Generative Retrieval over Dynamic Corpora
- How Do LLM-Generated Texts Impact Term-Based Retrieval Models?
- MemSyco-Bench: Benchmarking Sycophancy in Agent Memory
- Training with Missed Targets in Generative Recommendation: Separating Supervision from Probability Competition