INCARBench: A Benchmark for Scientific Configuration in VASP INCAR by Large Language Models
INCARBench evaluates Large Language Models' ability to handle scientific configuration files for VASP calculations. It tests models on generating and repairing INCAR files across various materials and workflows, focusing on physical meaning (semantic), numerical choices (policy), and overall task success (TCC). The results show gaps between parameter accuracy and achieving scientifically valid configurations.