ReproAgent: Contract-Guided Paper-to-Code Reproduction
cs.AI, cs.SE
Submitted: 2026-08-25
Updated: 2026-08-25
Comments: Accepted to Findings of EMNLP 2026
Code: https://github.com/kernel-14/ReproAgent
License: http://creativecommons.org/licenses/by/4.0/
The gist: Paper-to-code reproduction asks scientific AI agents to turn research papers into executable repositories that preserve the paper's method, protocol and artifacts.
Terminology
Abstract
Paper-to-code reproduction asks scientific AI agents to turn research papers into executable repositories that preserve the paper's method, protocol and artifacts. This is difficult because the specification is split: explicit paper content such as algorithms, metrics and artifacts is often lost across long agent trajectories, while implicit details such as framework defaults and conventions inherited from related work are absent from the paper. We introduce ReproAgent, a four-stage Prepare--Plan--Generate--Repair pipeline built around a persistent implementation contract with two channels: an implementation-requirement channel that turns paper snippets into code obligations, and a reference-evidence channel that retrieves content and structure evidence from related repositories. Both are bound to work packages, projected into file-level contracts, and consumed across generation and repair. On PaperBench Code-Dev, ReproAgent reaches the highest mean score among same-backbone scaffolds under both Claude-Sonnet-4.5 and Gemini-3-Flash. End-to-end channel ablations and per-paper cases support the contribution of both channels. Code and experimental artifacts are publicly available.
Sources
- SciCoQA: Quality Assurance for Scientific Paper--Code Alignment
- MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering
- Toward Autonomous Long-Horizon Engineering for ML Research
- ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models
- HiRAS: A Hierarchical Multi-Agent Framework for Paper-to-Code Generation and Execution
- GraphCodeAgent: Dual Graph-Guided LLM Agent for Retrieval-Augmented Repo-Level Code Generation
- DeepCode: Open Agentic Coding
- AutoP2C: An LLM-Based Agent Framework for Code Repository Generation from Multimodal Content in Academic Papers
- The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery
- Agent Laboratory: Using LLM Agents as Research Assistants
- ML-Bench: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Code
- SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Papers
- AutoReproduce: Automatic AI Experiment Reproduction with Paper Lineage
- Reflective Paper-to-Code Reproduction Enabled by Fine-Grained Verification
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems