Revisiting Text Ranking in Deep Research
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "Revisiting Text Ranking in Deep Research".
Title and authors: Tom: So, let's talk about who wrote this paper and what it’s called. It’s titled "Revisiting Text Ranking in Deep Research."
Jane: That title tells you right away that they aren't just looking at existing ranking methods; they are re-examining them specifically for the context of deep research agents.
Tom: Right. And the authors include Chuan Meng, Litu Ou, and Sean MacAvaney from Edinburgh and Glasgow universities.
Lu: They bring a strong background in both core text ranking theory and working with large language models, which should give them a solid foundation for this kind of evaluation.
Meng: I wonder if they focused too much on the academic setup and not enough on how these ranking methods actually behave when we put them into a production environment with real web search traffic.
Lalam: Based on what I see, they are setting up a framework to systematically check the effectiveness of these techniques across different scenarios.
The paper's summary: Jane: So, the summary explains that deep research agents use iterative reasoning and search to find answers, and the authors are providing a look at how established text ranking methods perform in this setup.
Tom: They highlight three specific areas they investigate: retrieval units, pipeline configurations involving re-rankers, and query characteristics.
Lu: The summary points out a key gap they are filling: we don't really know if the re-ranking methods we usually use actually give consistent benefits when the consumer is an LLM agent instead of a human.
Meng: That inconsistency sounds bad for deployment; you want predictable performance, not something that works well on paper but breaks in practice.
Lalam: They are also looking at how the queries issued by these agents—the queries they generate—might not match what was used to train the ranking models initially.
The paper's improvements: Tom: Now, let's get into what they actually suggest is better than what we currently know. They investigate retrieval units, pipeline configurations, and query characteristics as their main points of improvement.
Jane: For retrieval units, they found that passage-level information units are quite efficient for agents with limited context windows because their concise nature makes them more efficient under limited context-window budgets.
Lu: And on the other side, they saw that BM25, a lexical retriever, often outperforms neural retrievers when dealing with out-of-domain data compared to single-vector dense models.
Meng: That’s interesting because if BM25 is better and simpler to run at scale, that might suggest we should lean on it more for initial retrieval stages.
Lalam: They also focused heavily on re-ranking, showing it consistently lifts documents to the top of ranked lists and reduces the number of search calls needed by the agent.
Conclusion: Tom: So, to wrap up, they found that lexical retrievers like BM25 with specific settings can outperform neural rankers in most cases, and passage-level units help agents manage smaller context windows better.
Jane: They also proposed a query-to-question method to bridge the gap between agent queries and the training queries used for neural rankers, which seems important for improving their effectiveness.
Lu: The main thing they point out is that when an agent issues a search query, it can cause distribution drift for neural rankers, which limits their performance in deep research settings.
Meng: From my side, I see the Q2Q method as something that could stabilize performance during deployment because it handles that mismatch between the agent's real-time need and the model's training data better.
Lalam: They conclude that while established methods are still effective, addressing query mismatch and using passage-level units offers concrete ways to improve how these agents find information.
Chuan Meng, Litu Ou, Sean MacAvaney, Jeff Dalton
University of Edinburgh · University of Glasgow
cs.IR, cs.AI, cs.CL
Submitted: 2026-02-25
Updated: 2026-06-19
Comments: Accepted at the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2026)
Journal ref: Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2026), pp. 3006-3016
Code: https://github.com/ChuanMeng/text-ranking-in-deep-research
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 100/100
The gist: Deep research has emerged as an important task that aims to address hard queries that need extensive open-web exploration The paper examines key findings and best practices for text ranking methods…
Key concepts
- Passage-level Information Units
- These units focus on ranking individual passages rather than entire documents. They are efficient for agents with limited context windows because their concise nature allows for more search and reasoning iterations before hitting context limits, leading to higher accuracy when memory is constrained.
- Query-to-Question (Q2Q) Method
- This technique translates raw agent-issued web search queries into natural-language questions. This is crucial because the mismatch between the agent's query and the training queries used for neural rankers severely degrades performance; Q2Q helps bridge this gap by using a more effective question format.
- BM25 Retrieval
- BM25 is a lexical retrieval method that performs well, especially when agent queries follow a web-search style with keywords and phrases. Optimizing BM25 parameters, such as setting k1=10 and b=1 for document indexing, substantially improves performance by reducing sensitivity to length normalization.
- Re-ranking Efficacy
- A re-ranking stage consistently enhances ranking effectiveness and accuracy while simultaneously reducing the number of search calls. Deeper re-ranking depths further amplify these gains, confirming that this step is vital for deep research agents.
Terminology
Summary
Deep research has emerged as an important task that aims to address hard queries that need extensive open-web exploration The paper examines key findings and best practices for text ranking methods in deep research from three perspectives: retrieval units, pipeline configurations, and query characteristics
Research Gaps Addressed
The authors identify three research gaps in text ranking for deep research that have received limited attention First, passage-level information units have received little attention in deep research because most existing studies build agents that search and read at the document level Second, the understanding of re-ranking in deep research remains limited regarding whether widely-used re-ranking methods provide consistent benefits when the content consumer is an LLM-based agent Third, the potential mismatch between agent-issued queries and training queries used for text ranking methods remains underexplored
Retrieval Unit Effectiveness
RQ1 investigates how existing retrievers perform under passage-level and document-level retrieval units The findings show fivefold results regarding passage-level units Specifically, the concise nature of passage-level units enables more search and reasoning iterations before reaching context-window limits, resulting in higher answer accuracy than document-level units without a full-document reader Furthermore, BM25 on the passage corpus outperforms neural retrievers in most cases, achieving the highest accuracy of 0.572 across all retrieval settings in their study Learned sparse and multi-vector dense retrievers with only millions of parameters generalise better to web-search-style queries than 7B/8B single-vector dense models
Re-ranking Efficacy and Mismatch Mitigation
RQ2 addresses the effectiveness of a re-ranking stage in deep research under different initial retrievers, re-ranker types, and re-ranking cutoffs Re-ranking consistently improves ranking effectiveness and answer accuracy while reducing search calls, confirming its important role in deep research The BM25–monoT5-3B pipeline with gpt-oss-20b achieves the best results in their work, reaching 0.716 recall and 0. p. 1 Notably, the BM25–monoT5-3B pipeline
with gpt-oss-20b achieves the best results in our work, reaching
0. 716 recall and 0.Where the paper says it
Key Findings on Query Mismatch
The Q2Q (Q+R) variant consistently achieves significant improvements over raw agent-issued queries for both SPLADE-v3 and Qwen3-Embed-8B retrievers These results indicate that mismatch between agent-issued queries and the natural-language questions used to train neural rankers severely limits their effectiveness in deep research Q2Q (Q) provides little to no improvement over raw agent-issued queries, as illustrated by examples where the raw query is keyword-driven and under-specified Conversely, for BM25, Q2Q-generated questions even hurt performance, indicating that web-search-style queries are better suited to BM25 than natural-language questions
Conclusion and Contributions
The main contributions include reproducing a comprehensive set of text ranking methods in deep research They demonstrate that the lexical retriever BM25 with appropriate setup outperforms neural rankers in most cases, achieving the highest answer accuracy across all retrieval settings in their study They also find that passage-level units benefit agents with limited context windows and reduce sensitivity to document-length normalisation in lexical retrieval The authors propose the Q2Q method as a way to mitigate the training–inference query mismatch, which is critical for improving neural rankers in deep research They open-sourced their code, data, and all agent-generated traces at https://github.com/ChuanMeng/text-ranking-in-deep-research
Future Directions
Future directions identified include evaluating additional model families and larger model sizes to assess the generalisability of their findings Exploring additional rankers and configurations, such as scaling laws, is an important direction Extending deep research to conversational scenarios is also a promising avenue The paper was supported by a Turing AI Acceleration Fellowship funded by the Engineering and Physical Sciences Research Council (EPSRC), grant number EP/V025708/1
References
[1] Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, et al. 2025. gpt-oss-120b & gpt-oss-20b Model Card. p. 9 References
[1] Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin
Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, et al. gpt-oss-120b
& gpt-oss-20b Model Card. References
[1] Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin
Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, et al. gpt-oss-120b
& gpt-oss-20b Model Card. gpt-oss-120b
& gpt-oss-20b Model Card. arXiv preprint arXiv:2508. gpt-oss-120b
& gpt-oss-20b Model Card. arXiv preprint arXiv:2508. gpt-oss-120b
& gpt-oss-20b Model Card. arXiv preprint arXiv:2508. gpt-oss-120b
& gpt-oss-20b Model Card. arXiv preprint arXiv:2508. gpt-oss-120b
& gpt-oss-20b Model Card. arXiv preprint arXiv:2508. arXiv preprint arXiv:2508. 10925 (2025). arXiv preprint arXiv:2508. 10925 (2025).Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[4] Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary and Tong Wang. p. 9 Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. [4] Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong
Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir
Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. [4] Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong
Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir
Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. MS
MARCO: A Human Generated MAchine Reading COmprehension Dataset. [4] Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong
Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir
Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. MS
MARCO: A Human Generated MAchine Reading COmprehension Dataset. MS
MARCO: A Human Generated MAchine Reading COmprehension Dataset. arXiv
preprint arXiv:1611. MS
MARCO: A Human Generated MAchine Reading COmprehension Dataset. arXiv
preprint arXiv:1611. arXiv
preprint arXiv:1611. 09268 (2018). arXiv
preprint arXiv:1611. 09268 (2018).Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[5] Antoine Chaffin and Raphaël Sourty. p. 9 09268 (2018). [5] Antoine Chaffin and Raphaël Sourty. 09268 (2018). [5] Antoine Chaffin and Raphaël Sourty. [5] Antoine Chaffin and Raphaël Sourty. PyLate: Flexible Training and Retrieval
for Late Interaction Models. [5] Antoine Chaffin and Raphaël Sourty. PyLate: Flexible Training and Retrieval
for Late Interaction Models. PyLate: Flexible Training and Retrieval
for Late Interaction Models. 6334–6339.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[6] Zijian Chen, Xueguang Ma, Shengyao Zhuang, Ping Nie, Kai Zou, Andrew Liu, Joshua Green, Kshama Patel, Ruoxi Meng, Mingyi Su, et al. p. 9 6334–6339. [6] Zijian Chen, Xueguang Ma, Shengyao Zhuang, Ping Nie, Kai Zou, Andrew Liu,
Joshua Green, Kshama Patel, Ruoxi Meng, Mingyi Su, et al. [6] Zijian Chen, Xueguang Ma, Shengyao Zhuang, Ping Nie, Kai Zou, Andrew Liu,
Joshua Green, Kshama Patel, Ruoxi Meng, Mingyi Su, et al. BrowseCompPlus: A More Fair and Transparent Evaluation Benchmark of Deep-Research
Agent. arXiv preprint arXiv:2508. 06600 (2025).Where the paper says it
Where the paper says it
Where the paper says it
[7] Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M. p. 9 06600 (2025). [7] Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M. [7] Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M. Overview of the TREC 2019 Deep Learning Track. Overview of the TREC 2019 Deep Learning Track. In REC
2019.Where the paper says it
Where the paper says it
Where the paper says it
[8] Zhuyun Dai and Jamie Callan. p. 9 In REC
2019. [8] Zhuyun Dai and Jamie Callan. [8] Zhuyun Dai and Jamie Callan. Deeper Text Understanding for IR with
Contextual Neural Language Modeling.Where the paper says it
Where the paper says it
[10] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. p. 9 TREC CAsT 2021: The
Conversational Assistance Track Overview. [10] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. [10] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT:
Pre-training of Deep Bidirectional Transformers for Language Understanding. BERT:
Pre-training of Deep Bidirectional Transformers for Language Understanding. 4171–4186.Where the paper says it
Where the paper says it
Where the paper says it
[13] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi and et al. p. 9 2288–2292. [13] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin
Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. [13] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin
Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. DeepSeek-R1:
Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. DeepSeek-R1:
Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv
preprint arXiv:2501. arXiv
preprint arXiv:2501. 12948 (2025).Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[14] Ben He and Iadh Ounis. p. 9 12948 (2025). [14] Ben He and Iadh Ounis. [14] Ben He and Iadh Ounis. Term Frequency Normalisation Tuning for BM25
and DFR Models.Where the paper says it
Where the paper says it
[17] Tiansheng Hu, Yilun Zhao, Canyu Zhang, Arman Cohan, and Chen Zhao. p. 9 LoRA: Low-Rank Adaptation of Large Language
Models. [17] Tiansheng Hu, Yilun Zhao, Canyu Zhang, Arman Cohan, and Chen Zhao. [17] Tiansheng Hu, Yilun Zhao, Canyu Zhang, Arman Cohan, and Chen Zhao. SAGE: Benchmarking and Improving Retrieval for Deep Research Agents. jina-embeddings-v5-text:
Task-Targeted Embedding Distillation. arXiv preprint arXiv:2602. arXiv
preprint arXiv:2602. 05975 (2026).Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[18] Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan O Arik, Dong Wang, Hamed Zamani and Jiawei Han. p. 9 05975 (2026). [18] Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan O Arik, Dong Wang,
Hamed Zamani, and Jiawei Han. [18] Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan O Arik, Dong Wang,
Hamed Zamani, and Jiawei Han. Search-R1: Training LLMs to Reason and
Leverage Search Engines with Reinforcement Learning.Where the paper says it
Where the paper says it
[19] Marcin Kaszkiel and Justin Zobel. p. 9 Search-R1: Training LLMs to Reason and
Leverage Search Engines with Reinforcement Learning. [19] Marcin Kaszkiel and Justin Zobel. [19] Marcin Kaszkiel and Justin Zobel. Passage Retrieval Revisited. Passage Retrieval Revisited. In ACM
SIGIR Forum, Vol. In ACM
SIGIR Forum, Vol. ACM New York, NY, USA, 178–185.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[20] Omar Khattab and Matei Zaharia. p. 9 ACM New York, NY, USA, 178–185. [20] Omar Khattab and Matei Zaharia. [20] Omar Khattab and Matei Zaharia. Colbert: Efficient and Effective Passage
Search via Contextualized Late Interaction over BERT.Where the paper says it
Where the paper says it
[21] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin et al. p. 9 Colbert: Efficient and Effective Passage
Search via Contextualized Late Interaction over BERT. [21] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur
Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee,
et al. [21] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur
Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee,
et al. Natural Questions: A Benchmark for Question Answering Research. Natural Questions: A Benchmark for Question Answering Research. TACL 7 (2019), 453–466.Where the paper says it
Where the paper says it
Where the paper says it
[22] Carlos Lassance, Hervé Déjean, Thibault Formal, and Stéphane Clinchant. p. 9 TACL 7 (2019), 453–466. [22] Carlos Lassance, Hervé Déjean, Thibault Formal, and Stéphane Clinchant. [22] Carlos Lassance, Hervé Déjean, Thibault Formal, and Stéphane Clinchant. SPLADE-v3: New baselines for SPLADE. SPLADE-v3: New baselines for SPLADE. arXiv preprint arXiv:2403. arXiv preprint arXiv:2403. 06789 (2024).Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[31] Chuan Meng, Jiqun Liu, Mohammad Aliannejadi, Fengran Mo, Jeff Dalton and Maarten de Rijke. p. 9 Ranked List Truncation for Large Language Model-based
Re-Ranking. [31] Chuan Meng, Jiqun Liu, Mohammad Aliannejadi, Fengran Mo, Jeff Dalton,
and Maarten de Rijke. [31] Chuan Meng, Jiqun Liu, Mohammad Aliannejadi, Fengran Mo, Jeff Dalton,
and Maarten de Rijke. Re-Rankers as Relevance Judges. Re-Rankers as Relevance Judges. arXiv preprint
arXiv:2601. Re-Rankers as Relevance Judges. arXiv preprint
arXiv:2601. Re-Rankers as Relevance Judges. arXiv preprint
arXiv:2601. Re-Rankers as Relevance Judges. arXiv preprint
arXiv:2601. Re-Rankers as Relevance Judges. arXiv preprint
arXiv:2601. Re-Rankers as Relevance Judges. arXiv preprint
arXiv:2601. arXiv preprint
arXiv:2601. 04455 (2026).Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[38] Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. p. 9 04085 (2019). [38] Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. [38] Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. Document
Ranking with a Pretrained Sequence-to-Sequence Model.Where the paper says it
Where the paper says it
[45] Sahel Sharifymoghaddam and Jimmy Lin. p. 9 20595 (2025). [45] Sahel Sharifymoghaddam and Jimmy Lin. [45] Sahel Sharifymoghaddam and Jimmy Lin. Rerank Before You Reason:
Analyzing Reranking Tradeoffs through Effective Token Cost in Deep Search
Agents. arXiv preprint arXiv:2601. 14224 (2026).Where the paper says it
Where the paper says it
Where the paper says it
[58] Orion Weller, Kathryn Ricci, Eugene Yang, Andrew Yates, Dawn Lawrie and Benjamin Van Durme. p. 9 ArXiv (2023). [26] Jimmy Lin, Rodrigo Nogueira, and Andrew Yates. [39] Litu Ou, Kuan Li, Huifeng Yin, Liwen Zhang, Zhongwang Zhang, Xixi Wu,
Rui Ye, Zile Qiao, Pengjun Xie, Jingren Zhou, and Yong Jiang. BrowseConf: Confidence-Guided Test-Time Scaling for Web Agents. p. 9 [39] Litu Ou, Kuan Li, Huifeng Yin, Liwen Zhang, Zhongwang Zhang, Xixi Wu,
Rui Ye, Zile Qiao, Pengjun Xie, Jingren Zhou, and Yong Jiang. BrowseConf: Confidence-Guided Test-Time Scaling for Web Agents.Where the paper says it
Where the paper says it
Where the paper says it
[69] Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang and Dayiheng Liu et al. p. 9 05366 (2025). [25] Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan
Zhang. The series features fine-tuning of Qwen3
LLMs [62] on synthetic training data across multiple domains
and languages, generated by the Qwen3 models themselves. It
follows RepLLaMA to obtain query and document embeddings
and shares the same embedding dimension. gpt-oss-120b
& gpt-oss-20b Model Card. arXiv preprint arXiv:2508. arXiv preprint arXiv:2508. 10925 (2025).Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[71] Shengyao Zhuang, Houxing Ren, Linjun Shou, Jian Pei, Ming Gong, Guido Zuccon and Daxin Jiang. p. 9 6334–6339. [6] Zijian Chen, Xueguang Ma, Shengyao Zhuang, Ping Nie, Kai Zou, Andrew Liu,
Joshua Green, Kshama Patel, Ruoxi Meng, Mingyi Su, et al. 6334–6339. [6] Zijian Chen, Xueguang Ma, Shengyao Zhuang, Ping Nie, Kai Zou, Andrew Liu,
Joshua Green, Kshama Patel, Ruoxi Meng, Mingyi Su, et al. [33] Chuan Meng, Francesco Tonolini, Fengran Mo, Nikolaos Aletras, Emine Yilmaz,
and Gabriella Kazai. Bridging the Gap: From Ad-hoc to Proactive Search in
Conversations. [33] Chuan Meng, Francesco Tonolini, Fengran Mo, Nikolaos Aletras, Emine Yilmaz,
and Gabriella Kazai. Bridging the Gap: From Ad-hoc to Proactive Search in
Conversations. gpt-oss-120b
& gpt-oss-20b Model Card. arXiv preprint arXiv:2508. gpt-oss-120b
& gpt-oss-20b Model Card. arXiv preprint arXiv:2508. Trippas, and Svitlana Vakulenko. TREC CAsT 2022: Going Beyond User
Ask and System Retrieve with Initiative and Response Generation. Trippas, and Svitlana Vakulenko. TREC CAsT 2022: Going Beyond User
Ask and System Retrieve with Initiative and Response Generation.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[75] Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus and Amelia Glaese. 2025. BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents. p. 7 ,
the BrowseComp benchmark [55] includes queries that humans
typically cannot solve within ten minutes using search engines. Solving these queries typically requires LLM-based agents to perform multiple rounds of search and reasoning to gather, synthesise, ,
the BrowseComp benchmark [55] includes queries that humans
typically cannot solve within ten minutes using search engines. Solving these queries typically requires LLM-based agents to perform multiple rounds of search and reasoning to gather, synthesise, ReasonIR: Training Retrievers for Reasoning Tasks. arXiv preprint
arXiv:2504. ReasonIR: Training Retrievers for Reasoning Tasks. arXiv preprint
arXiv:2504. ReasonIR: Training Retrievers for Reasoning Tasks. arXiv preprint
arXiv:2504. arXiv preprint arXiv:2508. 10925 (2025). arXiv preprint arXiv:2508. 10925 (2025).Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[76] Yibo Wang, Lei Wang, Yue Deng, Keming Wu, Yao Xiao, Huanjin Yao, Liwei Kang, Hai Ye and Yongcheng Jing et al. p. 9 Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. [4] Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong
Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir
Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. CCS Concepts
• Information systems → Information retrieval. Keywords
Text ranking, Retrieval, Re-ranking, Deep research, Agentic search
ACM Reference Format:
Chuan Meng, Litu Ou, Sean MacAvaney, and Jeff Dalton. Revisiting
Text Ranking in Deep Research. In Proceedings of the 49th International
ACM SIGIR Conference on Research and Development in Information Retrieval
(SIGIR ’26), July 20–24, 2026, Melbourne, VIC, Australia.Where the paper says it
Where the paper says it
Where the paper says it
[77] Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie, Cunxiang Wang, Da Yin Hao Zeng et al. p. 9 Bridging the Gap: From Ad-hoc to Proactive Search in
Conversations. [34] Fengran Mo, Yifan Gao, Chuan Meng, Xin Liu, Zhuofeng Wu, Kelong Mao,
Zhengyang Wang, Pei Chen, Zheng Li, Xian Li, et al. Bridging the Gap: From Ad-hoc to Proactive Search in
Conversations. [34] Fengran Mo, Yifan Gao, Chuan Meng, Xin Liu, Zhuofeng Wu, Kelong Mao,
Zhengyang Wang, Pei Chen, Zheng Li, Xian Li, et al. [24] Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian
Zhang, and Zhicheng Dou. Search-o1: Agentic Search-Enhanced Large
Reasoning Models. arXiv preprint arXiv:2508. 10925 (2025). arXiv preprint arXiv:2508. 10925 (2025).Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[78] Peilin Zhou, Bruce Leon, Xiang Ying, Can Zhang, Yifan Shao, Qichen Ye, Dading Chong et al. p. 9 02592
(2025). [24] Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian
Zhang, and Zhicheng Dou. Recently, the IR community
has witnessed the emergence of deep research [46] scenarios. Deep
research aims to answer multi-hop, reasoning-intensive queries
that require extensive exploration of the open web and are difficult
for both humans and large language models (LLMs) [55]. arXiv preprint arXiv:2508. 10925 (2025).Where the paper says it
Where the paper says it
Where the paper says it
[79] Chuan Meng, Mohammad Aliannejadi and Maarten de Rijke. p. 9 2421–2425. [29] Chuan Meng, Mohammad Aliannejadi, and Maarten de Rijke. 2421–2425. [29] Chuan Meng, Mohammad Aliannejadi, and Maarten de Rijke. [29] Chuan Meng, Mohammad Aliannejadi, and Maarten de Rijke. System
Initiative Prediction for Multi-turn Conversational Information Seeking. [29] Chuan Meng, Mohammad Aliannejadi, and Maarten de Rijke. System
Initiative Prediction for Multi-turn Conversational Information Seeking. System
Initiative Prediction for Multi-turn Conversational Information Seeking. 1807–1817. System
Initiative Prediction for Multi-turn Conversational Information Seeking. 1807–1817.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[80] Fangyuan Xu, Rujun Han, Yanfei Chen, Zifeng Wang, I Hsu, Jun Yan, Vishy Tirumalashetty and Eunsol Choi. p. 9 05592
(2025). [48] Hongjin SU, Howard Yen, Mengzhou Xia, Weijia Shi, Niklas Muennighoff, Han yu
Wang, Liu Haisu, Quan Shi, Zachary S Siegel, Michael Tang, Ruoxi Sun, Jinsung
Yoon, Sercan O Arik, Danqi Chen, and Tao Yu. 05592
(2025). [48] Hongjin SU, Howard Yen, Mengzhou Xia, Weijia Shi, Niklas Muennighoff, Han yu
Wang, Liu Haisu, Quan Shi, Zachary S Siegel, Michael Tang, Ruoxi Sun, Jinsung
Yoon, Sercan O Arik, Danqi Chen, and Tao Yu. CCS Concepts
• Information systems → Information retrieval. Keywords
Text ranking, Retrieval, Re-ranking, Deep research, Agentic search
ACM Reference Format:
Chuan Meng, Litu Ou, Sean MacAvaney, and Jeff Dalton. CCS Concepts
• Information systems → Information retrieval. Keywords
Text ranking, Retrieval, Re-ranking, Deep research, Agentic search
ACM Reference Format:
Chuan Meng, Litu Ou, Sean MacAvaney, and Jeff Dalton. CCS Concepts
• Information systems → Information retrieval. Keywords
Text ranking, Retrieval, Re-ranking, Deep research, Agentic search
ACM Reference Format:
Chuan Meng, Litu Ou, Sean MacAvaney, and Jeff Dalton. Revisiting
Text Ranking in Deep Research. In Proceedings of the 49th International
ACM SIGIR Conference on Research and Development in Information Retrieval
(SIGIR ’26), July 20–24, 2026, Melbourne, VIC, Australia. Revisiting
Text Ranking in Deep Research. In Proceedings of the 49th International
ACM SIGIR Conference on Research and Development in Information Retrieval
(SIGIR ’26), July 20–24, 2026, Melbourne, VIC, Australia. Revisiting
Text Ranking in Deep Research. In Proceedings of the 49th International
ACM SIGIR Conference on Research and Development in Information Retrieval
(SIGIR ’26), July 20–24, 2026, Melbourne, VIC, Australia.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[9] Jeffrey Dalton, Chenyan Xiong, and Jamie Callan. p. 9 Deeper Text Understanding for IR with
Contextual Neural Language Modeling. [9] Jeffrey Dalton, Chenyan Xiong, and Jamie Callan. Deeper Text Understanding for IR with
Contextual Neural Language Modeling. [9] Jeffrey Dalton, Chenyan Xiong, and Jamie Callan. [9] Jeffrey Dalton, Chenyan Xiong, and Jamie Callan. TREC CAsT 2021: The
Conversational Assistance Track Overview. [9] Jeffrey Dalton, Chenyan Xiong, and Jamie Callan. TREC CAsT 2021: The
Conversational Assistance Track Overview.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[39] Litu Ou, Kuan Li, Huifeng Yin, Liwen Zhang, Zhongwang Zhang, Xixi Wu, Rui Ye et al. p. 9 Document
Ranking with a Pretrained Sequence-to-Sequence Model. [39] Litu Ou, Kuan Li, Huifeng Yin, Liwen Zhang, Zhongwang Zhang, Xixi Wu,
Rui Ye, Zile Qiao, Pengjun Xie, Jingren Zhou, and Yong Jiang. BrowseConf: Confidence-Guided Test-Time Scaling for Web Agents. arXiv preprint
arXiv:2510. arXiv preprint
arXiv:2510. 23458 (2025).Where the paper says it
Where the paper says it
Where the paper says it
[46] Zhengliang Shi, Yiqun Chen, Haitao Li, Weiwei Sun, Shiyu Ni, Yougang Lyu, Run-Ze Fan et al. p. 9 14224 (2026). [46] Zhengliang Shi, Yiqun Chen, Haitao Li, Weiwei Sun, Shiyu Ni, Yougang Lyu,
Run-Ze Fan, Bowen Jin, Yixuan Weng, Minjun Zhu, et al. [46] Zhengliang Shi, Yiqun Chen, Haitao Li, Weiwei Sun, Shiyu Ni, Yougang Lyu,
Run-Ze Fan, Bowen Jin, Yixuan Weng, Minjun Zhu, et al. Deep Research:
A Systematic Survey. Deep Research:
A Systematic Survey. arXiv preprint arXiv:2512. arXiv preprint arXiv:2512. 02038 (2025).Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[47] Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang and Ji-Rong Wen. p. 9 02038 (2025). [47] Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin
Zhao, Lei Fang, and Ji-Rong Wen. [47] Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin
Zhao, Lei Fang, and Ji-Rong Wen. R1-Searcher: Incentivizing the Search
Capability in LLMs via Reinforcement Learning. R1-Searcher: Incentivizing the Search
Capability in LLMs via Reinforcement Learning. arXiv preprint arXiv:2503. arXiv preprint arXiv:2503. 05592
(2025).Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[48] Hongjin SU, Howard Yen, Mengzhou Xia, Weijia Shi, Niklas Muennighoff, Han yu Wang, Liu Haisu et al. p. 9 05592
(2025). [48] Hongjin SU, Howard Yen, Mengzhou Xia, Weijia Shi, Niklas Muennighoff, Han yu
Wang, Liu Haisu, Quan Shi, Zachary S Siegel, Michael Tang, Ruoxi Sun, Jinsung
Yoon, Sercan O Arik, Danqi Chen, and Tao Yu. [48] Hongjin SU, Howard Yen, Mengzhou Xia, Weijia Shi, Niklas Muennighoff, Han yu
Wang, Liu Haisu, Quan Shi, Zachary S Siegel, Michael Tang, Ruoxi Sun, Jinsung
Yoon, Sercan O Arik, Danqi Chen, and Tao Yu. BRIGHT: A Realistic and
Challenging Benchmark for Reasoning-Intensive Retrieval.Where the paper says it
Where the paper says it
[51] Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava and Iryna Gurevych. 2021. BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models. p. 9 [6] Zijian Chen, Xueguang Ma, Shengyao Zhuang, Ping Nie, Kai Zou, Andrew Liu,
Joshua Green, Kshama Patel, Ruoxi Meng, Mingyi Su, et al. BrowseCompPlus: A More Fair and Transparent Evaluation Benchmark of Deep-Research
Agent.Where the paper says it
[55] Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung et al. p. 9 05592
(2025). [48] Hongjin SU, Howard Yen, Mengzhou Xia, Weijia Shi, Niklas Muennighoff, Han yu
Wang, Liu Haisu, Quan Shi, Zachary S Siegel, Michael Tang, Ruoxi Sun, Jinsung
Yoon, Sercan O Arik, Danqi Chen, and Tao Yu.Where the paper says it
[60] Fangyuan Xu, Rujun Han, Yanfei Chen, Zifeng Wang, I Hsu, Jun Yan, Vishy Tirumalashetty and Eunsol Choi. p. 9 05592
(2025). [48] Hongjin SU, Howard Yen, Mengzhou Xia, Weijia Shi, Niklas Muennighoff, Han yu
Wang, Liu Haisu, Quan Shi, Zachary S Siegel, Michael Tang, Ruoxi Sun, Jinsung
Yoon, Sercan O Arik, Danqi Chen, and Tao Yu.Where the paper says it
[71] Shengyao Zhuang, Houxing Ren, Linjun Shou, Jian Pei, Ming Gong, Guido Zuccon and Daxin Jiang. 2022. Bridging the Gap Between Indexing and Retrieval for Differentiable Search Index with Query Generation. arXiv preprint arXiv:2206.10128 (2022).
[77] Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie, Cunxiang Wang, Da Yin Hao Zeng et al. 2025. GLM-4.5: Agentic Reasoning and Coding (ARC) Foundation Models. p. 9 [24] Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian
Zhang, and Zhicheng Dou. Search-o1: Agentic Search-Enhanced Large
Reasoning Models.Where the paper says it
[79] Chuan Meng, Mohammad Aliannejadi and Maarten de Rijke. 2023. System Initiative Prediction for Multi-turn Conversational Information Seeking. In CIKM. 1807–1817.
[80] Fangyuan Xu, Rujun Han, Yanfei Chen, Zifeng Wang, I Hsu, Jun Yan, Vishy Tirumalashetty and Eunsol Choi. 2026. SAGE: Steerable Agentic Data Generation for Deep Search with Execution Feedback. arXiv preprint arXiv:2601.18202 (2026).
[9] Jeffrey Dalton, Chenyan Xiong, and Jamie Callan. 2021. TREC CAsT 2021: The Conversational Assistance Track Overview. In TREC.
Conclusion
Overall, text ranking methods developed by the IR community are still highly effective in deep research Several well-established findings continue to hold, including the effectiveness of BM25 with appropriate setup and the generalisation of BERT-based learned sparse and multi-vector dense retrievers over LLM-based single-vector dense retrievers However, web-search-style syntax of agent-issued queries induces distribution drift for neural rankers, limiting their effectiveness in deep research The proposed Q2Q method significantly mitigates this drift, improving neural ranking performance Passage-level units benefit agents with limited context windows and reduce sensitivity to document-length normalisation in lexical retrieval The paper was supported by a Turing AI Acceleration Fellowship funded by the Engineering and Physical Sciences Research Council (EPSRC), grant number EP/V025708/1
Acknowledgments
We thank Zijian Chen and Xueguang Ma for their guidance on using BrowseComp-Plus This research was supported by a Turing AI Acceleration Fellowship funded by the Engineering and Physical Sciences Research Council (EPSRC), grant number EP/V025708/1
[1] Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao et al. 2025. gpt-oss-120b & gpt-oss-20b Model Card. arXiv preprint arXiv:2508.10925 (2025).
[4] Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder et al. p. 9 Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. [4] Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong
Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir
Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang.Where the paper says it
[5] Antoine Chaffin and Raphaël Sourty. 2025. PyLate: Flexible Training and Retrieval for Late Interaction Models.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Passage-level units improve efficiency under limited context windows because
their concise nature makes them more efficient under limited context-window budgets.
This allows agents to performmore search and reasoning iterations before reaching context-window limits,
resulting inhigher answer accuracy
for agents with smaller contexts. -
Query-to-question (Q2Q) reformulation significantly reduces the mismatch between agent-issued queries and training queries, which is critical because
the mismatch between agent-issued queries and the training queries used for text ranking methods can severely degrade neural ranking effectiveness.
This is achieved by translating search queries intonatural-language questions
or including the reasoning trace to capturethe agent’s search intent.
-
Using document-oriented BM25 parameters, such as k1 = 10 and b = 1, substantially improves performance on documents by addressing sensitivity to length normalisation. This configuration can yield the
highest recall (0.647) across all settings in this section
for the gpt-oss-20b agent when indexing the full document. -
Re-ranking stages consistently improve ranking effectiveness and answer accuracy while reducing search calls, with
deeper re-ranking depths further amplifying these gains.
This suggests that increasing the number of reranking iterations from 10 to 50 provides further gains in recall and accuracy, alongside a reduction in search calls. -
The system can be optimized by employing retrieval units suited to agent output; for example,
lexical retrievers such as BM25
are particularly effective when agent-issued queries followa web-search style with keywords, phrases, and quotation marks for exact matching.
Abstract
Deep research has emerged as an important task that aims to address hard queries that need extensive open-web exploration. To tackle it, most prior work equips large language model (LLM)-based agents with opaque web search APIs, enabling agents to iteratively issue search queries, retrieve external evidence, and reason over it. Despite search's essential role in deep research, black-box web search APIs leave the behaviour of established text ranking methods in deep research largely unclear. To fill this gap, we reproduce key findings and best practices for text ranking methods in deep research. We examine their effectiveness from three perspectives: (i) retrieval units (documents vs. passages), (ii) pipeline configurations (different retrievers, re-rankers, and re-ranking depths), and (iii) query characteristics (the mismatch between agent-issued queries and the training queries of text rankers). We perform experiments on BrowseComp-Plus, a deep research dataset with a fixed corpus, evaluating 2 open-source agents, 5 retrievers, and 3 re-rankers. We find that agent-issued queries typically follow web-search-style syntax (e.g., quoted exact matches), favouring lexical, learned sparse, and multi-vector retrievers; passage-level units are more efficient under limited context windows, and avoid the difficulties of document length normalisation in lexical retrieval; re-ranking is highly effective. We further propose a query-to-question (Q2Q) method that translates agent-issued queries into natural language questions, significantly reducing the query mismatch.
Sources
- gpt-oss-120b & gpt-oss-20b Model Card
- jina-embeddings-v5-text: Task-Targeted Embedding Distillation
- MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
- BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent
- SPLADE v2: Sparse Lexical and Expansion Model for Information Retrieval
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- SAGE: Benchmarking and Improving Retrieval for Deep Research Agents
- SPLADE-v3: New baselines for SPLADE
- WebSailor: Navigating Super-human Reasoning for Web Agent
- Search-o1: Agentic Search-Enhanced Large Reasoning Models
- WebExplorer: Explore and Evolve for Training Long-Horizon Web Agents
- Re-Rankers as Relevance Judges
- Passage Re-ranking with BERT
- BrowseConf: Confidence-Guided Test-Time Scaling for Web Agents
- ReasonIR: Training Retrievers for Reasoning Tasks
- Rerank Before You Reason: Analyzing Reranking Tradeoffs through Effective Token Cost in Deep Search Agents
- Deep Research: A Systematic Survey
- R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning
- Tongyi DeepResearch Technical Report
- Llama 2: Open Foundation and Fine-Tuned Chat Models
Related papers
- GateSID: One Gate to Balance Semantic and Collaborative Signals across the Item Lifecycle
- SAM-D2Q: Aligning Multimodal Doc2Query with Search Demand and Conversion for E-commerce
- GINGER: Spicing Up Graph-Based Vector Search
- Subtraction Gets You More: Gap-Aware Retrieval for Multimodal Multi-Hop QA
- One-Step Retrieval Framework for Real-Time Sponsored Search Ads Using Hierarchical Text Representations
- CCFormer: Efficient Cross-Field Interaction and Hierarchical Sequence Compression for Industrial Recommendation at Tencent