A Methodological Analysis of Empirical Studies in Quantum Software Testing
quant-ph, cs.SE
Submitted: 2026-01-13
Updated: 2026-04-13
Journal ref: ACM Transactions on Software Engineering and Methodology, 2026
DOI: 10.1145/3819590
Code: https://github.com/NahidaNahida/QST-empirical-study
Project page: https://qiskit.github.io/qiskit-aer/apidocs/aer_provider.html
License: http://creativecommons.org/licenses/by/4.0/
The gist: In quantum software engineering (QSE), quantum software testing (QST) has attracted increasing attention as quantum software systems grow in scale and complexity.
Terminology
Abstract
In quantum software engineering (QSE), quantum software testing (QST) has attracted increasing attention as quantum software systems grow in scale and complexity. Since QST evaluates quantum programs through execution under designed test inputs, empirical studies are widely used to assess the effectiveness of testing approaches. However, the design and reporting of empirical studies in QST remain highly diverse, and a shared methodological understanding has yet to emerge, making it difficult to interpret results and compare findings across studies. This paper presents a methodological analysis of empirical studies in QST through a systematic examination of 59 primary studies identified from a literature pool of size 384. We organize our analysis around ten research questions that cover key methodological dimensions of QST empirical studies, including objects under test, baseline comparison, testing setup, experimental configuration, and tool and artifact support. Through cross-study analysis along these dimensions, we characterize current empirical practices in QST, identify recurring limitations and inconsistencies, and highlight open methodological challenges. Based on our findings, we derive insights and recommendations to inform the design, execution, and reporting of future empirical studies in QST.
Sources
- PennyLane: Automatic differentiation of hybrid quantum-classical computations
- VeriQBench: A Benchmark for Multiple Types of Quantum Circuits
- Equivalent Quantum Circuits
- QEF: Reproducible and Exploratory Quantum Software Experiments
- QCRMut: Quantum Circuit Random Mutant generator tool
- The Heisenberg Representation of Quantum Computers
- Evaluating Mutation-based Fault Localization for Quantum Programs
- NovaQ: Improving Quantum Program Testing through Diversity-Guided Test Case Generation
- Context-Aware Unit Testing for Quantum Subroutines
- QUT: A Unit Testing Framework for Quantum Subroutines
- Proq: Projection-based Runtime Assertions for Debugging on a Quantum Computer
- Testing Quantum Programs with Multiple Subroutines
- A Black-box Testing Framework for Oracle Quantum Programs
- Ablation Studies in Artificial Neural Networks
- The Cost of Certainty: Shot Budgets in Quantum Program Testing
- On the Feasibility of Quantum Unit Testing
- Quantum Algorithms
- Bloch Vector Assertions for Debugging Quantum Programs
- A Survey on Testing and Analysis of Quantum Software
- QuCheck: A Property-based Testing Framework for Quantum Programs in Qiskit
Related papers
- Reconquering Bell sampling on qudits: stabilizer learning and testing, quantum pseudorandomness bounds, and more
- Encrypted clones can leak: Classification of informative subsets in Quantum Encrypted Cloning
- Polynomial-time classical and quantum simulation of quantum impurity models
- A quasiparticle-protected superconducting qubit
- Theory of quantum-enhanced interferometry with general Markovian light sources
- A convergent hierarchy of spectral gap certificates for qubit Hamiltonians