ScholarCatalyst: A Benchmark for Retrieving Papers That Inspire New Research
cs.AI, cs.CL, cs.IR
Submitted: 2026-10-01
Updated: 2026-10-01
Comments: 57 pages
Code: https://github.com/stanford-iris-lab/ScholarCatalyst
Project page: https://yaoyichen.github.io/INF-X-Retriever
License: http://creativecommons.org/licenses/by/4.0/
The gist: What makes great scientists great? Even as AI systems start to make progress on open problems, scientists remain far ahead of them at sensing which prior idea, buried in an ever-growing archive of
Terminology
Abstract
What makes great scientists great? Even as AI systems start to make progress on open problems, scientists remain far ahead of them at sensing which prior idea, buried in an ever-growing archive of research, a new problem needs. To study this skill, we draw on researchers who know firsthand which earlier work advanced their completed projects, with papers serving as pointers to the ideas within. Using our automated pipeline that makes author annotation scalable, we build ScholarCatalyst by having 184 lead authors of 207 recent computer science papers label which candidates did or could have advanced their project, each with a detailed rationale. We introduce a retrieval task with author-provided judgments: given an initial research question, retrieve these papers from only the literature available when the project began. Agentic search does no better than embedding retrieval (0.42 vs. 0.48 Recall@20) despite calling that same retriever as a tool. Even an agent built on Claude Fable 5.1, which may have seen the completed papers during training, reaches only 0.51 R@20. These results highlight the need for new training recipes that equip models with expert intuition for searching broad corpora. We envision ScholarCatalyst as a step toward scientific agents that can take a half-formed idea and point to the prior research it needs.
Sources
- OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented LMs
- Are we done with ImageNet?
- ReasonEmbed: Enhanced Text Embeddings for Reasoning-Intensive Document Retrieval
- Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
- HTML papers on arXiv -- why it is important, and how we made it happen
- Approaching Human-Level Forecasting with Language Models
- GIANTS: Generative Insight Anticipation from Scientific Literature
- Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning
- The Semantic Scholar Open Data Platform
- Towards General Text Embeddings with Multi-stage Contrastive Learning
- Beyond Semantic Similarity: Rethinking Retrieval for Agentic Search via Direct Corpus Interaction
- Harnessing Large Language Models for Scientific Novelty Detection
- DIVER: A Multi-Stage Approach for Reasoning-intensive Information Retrieval
- CSFCube -- A Test Collection of Computer Science Research Articles for Faceted Query by Example
- PaperScout: An Autonomous Agent for Academic Paper Search with Process-Aware Sequence-Level Policy Optimization
- Chain of Retrieval: Multi-Aspect Iterative Search Expansion and Post-Order Search Aggregation for Full Paper Retrieval
- OpenAlex: A fully-open index of scholarly works, authors, venues, institutions, and concepts
- ImageNet Large Scale Visual Recognition Challenge
- Is Grep All You Need? How Agent Harnesses Reshape Agentic Search
- Gemini Embedding 2: A Native Multimodal Embedding Model from Gemini
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems