Evaluating Instrument-Centered AI Agents for Table-Top Science Experiments
cond-mat.mes-hall
Submitted: 2026-10-30
Updated: 2026-10-30
Comments: 36 pages (26 main text + 10 supplemental material), 14 figures, 5 tables. Code: https://github.com/caidish/instrMCP and https://github.com/caidish/qdevbench
Code: https://github.com/caidish/instrMCP
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Terminology
Sources
- Software Engineering as Instrumentation for the Long Tail of Scientific Software
- SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
- ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
- Large Language Models are not Fair Evaluators
Related papers
- Coexistence of polariton bound states in the continuum and 2D radiative excitons
- Hidden Quantum-Geometric Connection between Injection and Shift Currents
- Specific absorption rate of uniaxial single-domain nanomagnets: stochastic spin dynamics versus linear response theory
- The ideal limit of rhombohedral graphene: Dynamical skyrmion lattices and their excitations
- Valley polarization of chiral excitonic bound states induced by band geometry
- On-Device Control of Electronic Friction