Revisiting Text Ranking in Deep Research
summary
The gist
Deep research has emerged as an important task that aims to address hard queries that need extensive open-web exploration The paper examines key findings and best practices for text ranking methods…
In short
The research explores text ranking methods for deep research agents by examining retrieval units, re-ranking pipelines, and query characteristics. Findings show passage-level units are efficient for limited contexts, BM25 excels in lexical retrieval, and a Query-to-Question (Q2Q) method mitigates mismatches between agent queries and training data to boost neural ranker performance.
Key concepts
- Passage-level Information Units
- These units focus on ranking individual passages rather than entire documents. They are efficient for agents with limited context windows because their concise nature allows for more search and reasoning iterations before hitting context limits, leading to higher accuracy when memory is constrained.
- Query-to-Question (Q2Q) Method
- This technique translates raw agent-issued web search queries into natural-language questions. This is crucial because the mismatch between the agent's query and the training queries used for neural rankers severely degrades performance; Q2Q helps bridge this gap by using a more effective question format.
- BM25 Retrieval
- BM25 is a lexical retrieval method that performs well, especially when agent queries follow a web-search style with keywords and phrases. Optimizing BM25 parameters, such as setting k1=10 and b=1 for document indexing, substantially improves performance by reducing sensitivity to length normalization.
- Re-ranking Efficacy
- A re-ranking stage consistently enhances ranking effectiveness and accuracy while simultaneously reducing the number of search calls. Deeper re-ranking depths further amplify these gains, confirming that this step is vital for deep research agents.
Terminology used across episodes
This episode discusses
- Revisiting Text Ranking in Deep Research · Paper Radio
- gpt-oss-120b & gpt-oss-20b Model Card
- jina-embeddings-v5-text: Task-Targeted Embedding Distillation
- MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
- BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent
- SPLADE v2: Sparse Lexical and Expansion Model for Information Retrieval
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- SAGE: Benchmarking and Improving Retrieval for Deep Research Agents
- SPLADE-v3: New baselines for SPLADE
- WebSailor: Navigating Super-human Reasoning for Web Agent
- Search-o1: Agentic Search-Enhanced Large Reasoning Models
- WebExplorer: Explore and Evolve for Training Long-Horizon Web Agents
- Re-Rankers as Relevance Judges
- Passage Re-ranking with BERT
- BrowseConf: Confidence-Guided Test-Time Scaling for Web Agents
- ReasonIR: Training Retrievers for Reasoning Tasks
- Rerank Before You Reason: Analyzing Reranking Tradeoffs through Effective Token Cost in Deep Search Agents
- Deep Research: A Systematic Survey
- R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning
- Tongyi DeepResearch Technical Report
- Llama 2: Open Foundation and Fine-Tuned Chat Models
The paper
Revisiting Text Ranking in Deep Research · Read on arXiv
Chuan Meng, Litu Ou, Sean MacAvaney, Jeff Dalton
University of Edinburgh · University of Glasgow
Deep research has emerged as an important task that aims to address hard queries that need extensive open-web exploration. To tackle it, most prior work equips large language model (LLM)-based agents with opaque web search APIs, enabling agents to iteratively issue search queries, retrieve external evidence, and reason over it. Despite search's essential role in deep research, black-box web search APIs leave the behaviour of established text ranking methods in deep research largely unclear. To fill this gap, we reproduce key findings and best practices for text ranking methods in deep research. We examine their effectiveness from three perspectives: (i) retrieval units (documents vs. passages), (ii) pipeline configurations (different retrievers, re-rankers, and re-ranking depths), and (iii) query characteristics (the mismatch between agent-issued queries and the training queries of text rankers). We perform experiments on BrowseComp-Plus, a deep research dataset with a fixed corpus, evaluating 2 open-source agents, 5 retrievers, and 3 re-rankers. We find that agent-issued queries typically follow web-search-style syntax (e.g., quoted exact matches), favouring lexical, learned sparse, and multi-vector retrievers; passage-level units are more efficient under limited context windows, and avoid the difficulties of document length normalisation in lexical retrieval; re-ranking is highly effective. We further propose a query-to-question (Q2Q) method that translates agent-issued queries into natural language questions, significantly reducing the query mismatch.
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "Revisiting Text Ranking in Deep Research".
Title and authors: Tom: So, let's talk about who wrote this paper and what it’s called. It’s titled "Revisiting Text Ranking in Deep Research."
Jane: That title tells you right away that they aren't just looking at existing ranking methods; they are re-examining them specifically for the context of deep research agents.
Tom: Right. And the authors include Chuan Meng, Litu Ou, and Sean MacAvaney from Edinburgh and Glasgow universities.
Lu: They bring a strong background in both core text ranking theory and working with large language models, which should give them a solid foundation for this kind of evaluation.
Meng: I wonder if they focused too much on the academic setup and not enough on how these ranking methods actually behave when we put them into a production environment with real web search traffic.
Lalam: Based on what I see, they are setting up a framework to systematically check the effectiveness of these techniques across different scenarios.
The paper's summary: Jane: So, the summary explains that deep research agents use iterative reasoning and search to find answers, and the authors are providing a look at how established text ranking methods perform in this setup.
Tom: They highlight three specific areas they investigate: retrieval units, pipeline configurations involving re-rankers, and query characteristics.
Lu: The summary points out a key gap they are filling: we don't really know if the re-ranking methods we usually use actually give consistent benefits when the consumer is an LLM agent instead of a human.
Meng: That inconsistency sounds bad for deployment; you want predictable performance, not something that works well on paper but breaks in practice.
Lalam: They are also looking at how the queries issued by these agents—the queries they generate—might not match what was used to train the ranking models initially.
The paper's improvements: Tom: Now, let's get into what they actually suggest is better than what we currently know. They investigate retrieval units, pipeline configurations, and query characteristics as their main points of improvement.
Jane: For retrieval units, they found that passage-level information units are quite efficient for agents with limited context windows because their concise nature makes them more efficient under limited context-window budgets.
Lu: And on the other side, they saw that BM25, a lexical retriever, often outperforms neural retrievers when dealing with out-of-domain data compared to single-vector dense models.
Meng: That’s interesting because if BM25 is better and simpler to run at scale, that might suggest we should lean on it more for initial retrieval stages.
Lalam: They also focused heavily on re-ranking, showing it consistently lifts documents to the top of ranked lists and reduces the number of search calls needed by the agent.
Conclusion: Tom: So, to wrap up, they found that lexical retrievers like BM25 with specific settings can outperform neural rankers in most cases, and passage-level units help agents manage smaller context windows better.
Jane: They also proposed a query-to-question method to bridge the gap between agent queries and the training queries used for neural rankers, which seems important for improving their effectiveness.
Lu: The main thing they point out is that when an agent issues a search query, it can cause distribution drift for neural rankers, which limits their performance in deep research settings.
Meng: From my side, I see the Q2Q method as something that could stabilize performance during deployment because it handles that mismatch between the agent's real-time need and the model's training data better.
Lalam: They conclude that while established methods are still effective, addressing query mismatch and using passage-level units offers concrete ways to improve how these agents find information.
More episodes
- 2505.16254-Human-AI Writing as Distributed Cognition: Reassessing Collaborative Writing Theories in the Age of LLMs
- 2506.10913-Choreographic Quick Changes: First-Class Location (Set) Polymorphism
- 2507.14812-A Black-Box Approach for Exogenous Replenishment in Online Resource Allocation
- 2507.15448-Frames over finite fields with respect to the Galois inner product and their connections to self-orthogonal and LCD codes
- 2508.21572-NewsReX: An Open-Source Multi-Framework for Neural News Recommendation
- 2509.17028-Impact of Packetization on Network Calculus Analysis
- 2510.17342-AoA Services in 5G Networks: A Framework for Real-World Implementation and Systematic Testing
- 2511.04088-Efficient and rate-optimal list-decoding in the presence of minimal feedback
- 2511.17264-Notes on Stack Machines and Quantum Stack Machines
- 2601.15318-On the closest balanced game