Read the Paper, Write the Code: Agentic Reproduction of Social-Science Results

arXiv:2604.21965 · cs.AI · Submitted 2026-04-23 · Read on arXiv

cs.AI

Submitted: 2026-04-23

Updated: 2026-09-29

Comments: This extended version of the paper forthcoming at EMNLP provides additional detail and discussion for a social science audience

Code: https://github.com/benjamin-kohler/social_science_replicability

Project page: https://firebench.github.io

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Papers — 4× in this paper · explained in 3 episodes
• Trace — 1× in this paper · explained in 2 episodes

Sources

• ResearchCodeBench: Benchmarking LLMs on Implementing Novel Machine Learning Research Code
• LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods
• The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery
• ReplicatorBench: Benchmarking LLM Agents for Replicability in Social and Behavioral Sciences
• Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning
• Automating Computational Reproducibility in Social Science: Comparing Prompt-Based and Agent-Based Approaches
• Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers
• CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark
• PaperBench: Evaluating AI's Ability to Replicate AI Research
• GLM-5: from Vibe Coding to Agentic Engineering
• SciCode: A Research Coding Benchmark Curated by Scientists
• Scaling Reproducibility: An AI-Assisted Workflow for Large-Scale Replication and Reanalysis
• SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
• PaperRepro: Automated Computational Reproducibility Assessment for Social Science Papers
• AutoReproduce: Automatic AI Experiment Reproduction with Paper Lineage

Related papers

• MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
• Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
• Agents' Last Exam
• The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
• MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
• Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems