Publications as benchmarks: auditing AI agents on spatial transcriptomics alignment
q-bio.GN, cs.AI, stat.AP
Submitted: 2026-11-09
Updated: 2026-11-09
Code: https://github.com/Y-Chen-LAB/spatial-alignment-agent
Project page: https://ericzhy0815.github.io/agent-trace-explorer
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
The gist: AI agents are increasingly automating data analysis, but responsible adoption requires reliable performance evaluation, which is costly in specialized scientific domains.
Terminology
Abstract
AI agents are increasingly automating data analysis, but responsible adoption requires reliable performance evaluation, which is costly in specialized scientific domains. We introduce a framework that translates scientific publications into auditable benchmarks for AI agents and demonstrate its use on spatial transcriptomics alignment tasks. Across 121 tasks, 16 baseline methods, and six AI agents, the best agents (GPT-6-Astra and Gemini-4-Argon) matched or outperformed all baselines on 80% of tasks, often using surprisingly simple heuristics. Agent performance was robust to prompting and access to pre-installed packages. Benchmark construction also uncovered errors in prior work and shortcuts in current evaluation metrics. Finally, we show that agent-generated solutions can be distilled into a reusable program that ranks in the top 10% of baseline methods on unseen data. Our framework offers a scalable way to evaluate AI agents in specialized scientific domains by leveraging and stress-testing existing papers and metrics.
Sources
- The More You Automate, the Less You See: Hidden Pitfalls of AI Scientist Systems
- Natural Emergent Misalignment from Reward Hacking in Production RL
- Establishing Best Practices for Building Rigorous Agentic Benchmarks
- Evaluating Large Language Models Trained on Code
- Program Synthesis with Large Language Models
- Paper2Agent: Reimagining Research Papers As Interactive and Reliable AI Agents
- NatureBench: Can Coding Agents Match the Published SOTA of Nature-Family Papers?
- SpatialBench: Can Agents Analyze Real-World Spatial Biology Data?
Related papers
- SeekRBP: Leveraging Sequence-Structure Integration with Reinforcement Learning for Receptor-Binding Protein Identification
- Genome-Factory: A Library for Tuning, Deploying, and Interpreting Genomic Foundation Models
- AVA: Agentic Variant Adjudication for Automated, Auditable Read-Level Review in Cancer Diagnostic Sequencing
- C3P: Contrastive promoter-protein pretraining yields representations capturing bacterial gene regulation
- Leveraging genomic deep learning models for the prediction of non-coding variant effects
- Adaptive Allele-Haplotype Incidence Indexing for Pangenome Queries: Representation, Reconstruction, and Benchmarking