The Implementation Lottery: Auditing Idea Reliability in Automated Research
cs.MA, cs.AI
Submitted: 2026-07-29
Updated: 2026-10-04
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Terminology
Sources
- Many AI Analysts, One Dataset: Navigating the Agentic Data Science Multiverse
- The Benchmark Lottery
- ResearchGym: Evaluating Language Model Agents on Real-World AI Research
- The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery
- Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation
- WhatWorkedBench: Benchmarking Experimental Understanding in AI Agents
- Auto Research with Specialist Agents Develops Effective and Non-Trivial Training Recipes
- Closed-loop Auto Research for Molecular Property Prediction: Discovering and Certifying Generalizable Improvements
- Auto Research for Materials: Auditable AI-Scientist Workflows with Held-Out Transfer
- Scores Alone Do Not Prove Discovery: The Discovery Certification Protocol for Auditing AI Research Agents
- AlphaEvolve: A coding agent for scientific and algorithmic discovery
- The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search
Related papers
- Highway Congestion Reduction through Reinforcement Learning Based Eulerian Headway Control
- ORACLE-SWE: Quantifying the Contribution of Critical Information Signals on SWE Agents
- You Only Align Once: Propagating Cooperative Behaviors in Multi-Agent Systems through Seed Agents
- Deny Without Disabling: Authorization-Paired Evaluation and Control for Multi-Agent Systems
- MA-SAPO: Multi-Agent Reasoning for Score-Aware Prompt Optimization
- PeroMAS: A Multi-agent System of Perovskite Material Discovery