Can Coding Agents Reproduce Findings in Computational Materials Science?

arXiv:2605.00803 · cs.CL, cond-mat.mtrl-sci, cs.AI, cs.SE · Submitted 2026-05-01 · Read on arXiv

cs.CL, cond-mat.mtrl-sci, cs.AI, cs.SE

Submitted: 2026-05-01

Updated: 2026-09-01

Comments: COLM 2026. The second and third authors contributed equally to this work

Code: https://github.com/JHU-CLSP/AutoMathttps:

Project page: https://jhu-clsp.github.io/AutoMathttps://github.com/JHU-CLSP/AutoMathttps://hf.co/datasets/jhu-clsp/AutoMat

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Benchmark — 37× in this paper · explained in 2 episodes
• Accuracy — 20× in this paper · explained in 1 episode
• Claude Code — 14× in this paper
• Verification — 10× in this paper · explained in 2 episodes
• Density Functional Theory — 9× in this paper
• Fidelity — 8× in this paper · explained in 2 episodes
• Molecular Dynamics — 8× in this paper
• Core — 7× in this paper · explained in 2 episodes
• Trace — 7× in this paper · explained in 2 episodes
• Density Functional Theory (DFT) — 5× in this paper · explained in 2 episodes
• Failure Modes — 5× in this paper · explained in 3 episodes
• Agentic System — 4× in this paper · explained in 2 episodes
• Benchmarking — 3× in this paper · explained in 3 episodes
• Calibration — 3× in this paper · explained in 7 episodes
• Discrete Dislocation Dynamics — 3× in this paper
• Evaluation Dimensions — 3× in this paper · explained in 3 episodes
• Foundation Model — 3× in this paper · explained in 3 episodes
• Foundation Models — 3× in this paper · explained in 7 episodes
• Provenance — 3× in this paper · explained in 3 episodes
• Adaptation — 2× in this paper · explained in 1 episode
• Consistency — 2× in this paper · explained in 3 episodes
• DFT Calculations — 2× in this paper · explained in 2 episodes
• LLM judge — 2× in this paper · explained in 2 episodes
• Ablation — 1× in this paper · explained in 1 episode
• Ablation study — 1× in this paper · explained in 1 episode
• Active Learning — 1× in this paper · explained in 6 episodes
• Agentic AI — 1× in this paper · explained in 1 episode
• Agentic AI systems — 1× in this paper · explained in 1 episode
• Agentic Coding — 1× in this paper · explained in 1 episode
• Defense Advanced Research Projects Agency — 1× in this paper
• F1 Score — 1× in this paper · explained in 3 episodes
• Failure Taxonomy — 1× in this paper · explained in 2 episodes
• Large Language Models (LLMs) — 1× in this paper · explained in 6 episodes
• Nudged Elastic Band — 1× in this paper

Sources

• Evaluating Large Language Models Trained on Code
• Kimi K2.5: Visual Agentic Intelligence
• MatTools: Benchmarking Large Language Models for Materials Science Tools

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering