IdeaScientist: Orchestrating Agents for Grounded Scientific Ideation
cs.CL
Submitted: 2026-10-02
Updated: 2026-10-09
Code: https://github.com/jiarui-liu/IdeaScientist
License: http://creativecommons.org/licenses/by/4.0/
The gist: Despite rapid progress in automating scientific research, generating promising and well grounded research solutions remains a central challenge.
Abstract
Despite rapid progress in automating scientific research, generating promising and well grounded research solutions remains a central challenge. We isolate research ideation as a standalone task and build our solution on the intuition that a challenge in one field can often be addressed by a mechanism that solved an analogous challenge in another. Accordingly, we introduce IdeaScientist, which decomposes ideation into gap finding, innovation, and report writing, and trains each role with reinforcement learning. These roles identify limitations in related work, draw solution intuitions from analogous problem settings, and develop those intuitions into complete research proposals. To facilitate discovery of insights across domains, we construct the Svalbard Idea Vault, a corpus of 2.77M decomposed research ideas for retrieval, training, and temporally controlled evaluation. Our evaluation restricts access to literature available before a cutoff date and assesses how closely proposed directions align with those later explored in 15K papers authored by human researchers. On Qwen3.6-27B, IdeaScientist outperforms the strongest open-source autoresearch baseline by 14.0%, driven mainly by gains in novelty. On this 27B open backbone, IdeaScientist even outperforms Claude Code SDK with Claude-4.8-Opus and Codex SDK with GPT-5.4, by up to 5.9%.
Sources
- Pre-review to Peer review: Pitfalls of Automating Reviews using Large Language Models
- HealthBench: Evaluating Large Language Models Towards Improved Human Health
- How Closely Do LLM Reviews Align with Human Peer Review?
- ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning
- Ideation Arena: Evaluating LLM Generated Research Ideas with Battle-style Human Expert Assessment
- Read, Grep, and Synthesize: Diagnosing Cross-Domain Seed Exposure for LLM Research Ideation
- CoCoReviewBench: A Completeness- and Correctness-Oriented Benchmark for AI Reviewers
- Training AI Scientists to Replicate Research
- AI Research Preference Models
- Training AI Co-Scientists Using Rubric Rewards
- SoundnessBench: Can Your AI Scientist Really Tell Good Research Ideas from Bad Ones?
- Nova: An Iterative Planning and Search Approach to Enhance Novelty and Diversity of LLM Generated Ideas
- Reinforcement Learning with Rubric Anchors
- Style Wins, Substance Loses: A Diagnosis of LLM-as-Judge in Idea Generation
- HindSight: Evaluating LLM-Generated Research Ideas via Future Impact
- Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning
- Toward Generalist Autonomous Research via Hypothesis-Tree Refinement
- LLM-as-a-Reviewer: Benchmarking Their Ability, Divergence, and Prompt Injection Resistance as Paper Reviewers
- How Can Rhetoric Reward-Hack AI Reviewers? Dissecting Rhetorical Sensitivity in AI-Based Peer Review
- Chain-of-Agents: End-to-End Agent Foundation Models via Multi-Agent Distillation and Agentic RL
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering