QC-Stark: A Multi-Task Benchmark Revealing Capability Dissociations in LLMs Evaluated on Quantum Computing Tasks
quant-ph, cs.AI, cs.LG
Submitted: 2026-09-28
Updated: 2026-09-28
Comments: accepted at the Quantum AI Workshop, Indianapolis IN, August 2026
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
The gist: We introduce QC-Stark, a benchmark for evaluating large language models (LLMs) on 11 quantum computing (QC) tasks, spanning circuit construction, debugging, compilation, error correction, and
Terminology
Abstract
We introduce QC-Stark, a benchmark for evaluating large language models (LLMs) on 11 quantum computing (QC) tasks, spanning circuit construction, debugging, compilation, error correction, and simulation. Across 2,750 evaluations (10 models times 11 tasks x 5 difficulty levels x 5 seeds), we find that overall rankings mask substantial per-task variation. The Spearman correlation between overall and per-task rankings is statistically insignificant for 4 out of the 11 tasks included in this benchmark. A 2-parameter Item Response Theory (IRT) model validates measurement quality, and prompt sensitivity analysis confirms ranking robustness across prompt conditions. All tasks are auto-verifiable via execution, thus not requiring any manual evaluation. We make the code and data publicly available on Huggingface.
Sources
- Qiskit QuantumKatas: Adapting Microsoft's Quantum Computing exercises for LLM evaluation
- Quantum-Audit: Evaluating the Reasoning Limits of LLMs on Quantum Computing
- QCircuitBench: A Large-Scale Dataset for Benchmarking Quantum Algorithm Design
- SciCode: A Research Coding Benchmark Curated by Scientists
- CMT-Benchmark: A Benchmark for Condensed Matter Theory Built by Expert Researchers
- Holistic Evaluation of Language Models
- Qiskit HumanEval: An Evaluation Benchmark For Quantum Code Generative Models
- Are Emergent Abilities of Large Language Models a Mirage?
- Tulu 3: Pushing Frontiers in Open Language Model Post-Training
Related papers
- Reconquering Bell sampling on qudits: stabilizer learning and testing, quantum pseudorandomness bounds, and more
- Encrypted clones can leak: Classification of informative subsets in Quantum Encrypted Cloning
- Polynomial-time classical and quantum simulation of quantum impurity models
- A quasiparticle-protected superconducting qubit
- Theory of quantum-enhanced interferometry with general Markovian light sources
- A convergent hierarchy of spectral gap certificates for qubit Hamiltonians