Science sandboxes measure the scientific capability of AI agents
q-bio.QM, cs.AI
Submitted: 2026-08-31
Updated: 2026-08-31
Comments: 72 pages, 5 main figures, 3 tables, and 5 supplementary figures; includes supplementary agent instructions and harness
Code: https://github.com/asr2210/science-sandbox
Project page: https://openai.com/index/introducing-gpt-5-5
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
The gist: Scientific progress depends not only on finding solutions, but on learning the rules that explain why they work and using that understanding to design better experiments.
Terminology
Abstract
Scientific progress depends not only on finding solutions, but on learning the rules that explain why they work and using that understanding to design better experiments. We introduce science sandboxes, a framework for studying this capability in AI agents through repeated cycles of experimentation, feedback, and hypothesis revision. Science sandboxes invite an agent to query the natural world in different ways, ranging from "wet" physical experiments, to "damp" predictive models trained on empirical data, to "dry" invented rules. By establishing a common experimental loop and a protocol for evaluating agents within it, science sandboxes allow assessment of both quantitative performance on specific metrics and qualitative scientific reasoning, across a spectrum of empirical verifiability. Here, we instantiate this framework in two biological settings, models of regulatory genomics and protein fitness prediction, and examine the capabilities of frontier agents. Across these settings, we could see when agents successfully optimized a quantitative metric without understanding the rules underlying the system. In particular, their scientific reasoning deteriorated when they encountered systems whose rules fell outside familiar biological priors. By highlighting such failure modes, science sandboxes make the frontier of scientific capability measurable and provide a controlled setting in which to study and ultimately expand it.
Sources
- Empowering Biomedical Discovery with AI Agents
- DiscoveryBench: Towards Data-Driven Discovery with Large Language Models
- LAB-Bench: Measuring Capabilities of Language Models for Biology Research
- PaperBench: Evaluating AI's Ability to Replicate AI Research
- BixBench: a Comprehensive Benchmark for LLM-based Agents in Computational Biology
- Iterative Foundation Model Fine-Tuning on Multiple Rewards
- Reward-Guided Iterative Refinement in Diffusion Models at Test-Time with Applications to Protein and DNA Design
- Life After Benchmark Saturation: A Case Study of CORE-Bench
Related papers
- A likelihood-based framework for simultaneously learning both noise and growth dynamics using biologically-informed neural networks
- Automated Lesion Segmentation of Stroke MRI Using nnU-Net: A Comprehensive External Validation Across Acute and Chronic Lesions
- Imaging-anchored multiomics in cardiovascular disease: integrating cardiac imaging, bulk, single-cell, and spatial transcriptomics
- Resolving satellite-in situ mismatches in Net Primary Production using high-frequency in situ bio-optical observations in the subpolar Northwest Atlantic
- easyplater: The easy way to generate microplate designs deconvolved from multivariate clinical data
- Essential Workers at Risk: An Agent-Based Model (SAFE-ABM) with Bayesian Uncertainty Quantification