QBugLM: An Agentic Benchmarking Framework for LLM-based Quantum Software Debugging
cs.SE, cs.ET, quant-ph
Submitted: 2026-06-05
Updated: 2026-06-05
Comments: This paper was accepted at IEEE QSW 2026
DOI: 10.1109/QSW72780.2026.00028
Code: https://github.com/qachub/qbuglm
License: http://creativecommons.org/licenses/by/4.0/
The gist: Quantum software bugs often yield silent, incorrect outputs rather than explicit errors, making them particularly difficult to detect and repair with conventional techniques.
Terminology
Abstract
Quantum software bugs often yield silent, incorrect outputs rather than explicit errors, making them particularly difficult to detect and repair with conventional techniques. Although large language models (LLMs) have shown strong performance on classical software engineering tasks, their ability to debug quantum code remains largely unexplored. To bridge this gap, we propose QBugLM, a multi-agent framework that automates the quantum software debugging pipeline, from taxonomy-driven bug injection to LLM-based detection and repair, and finally to simulation-based validation, for framework-agnostic OpenQASM 3.0 programs. We further conduct a comprehensive case study using QBugLM to benchmark two LLMs, Claude 4.6 Sonnet and Qwen3 Coder Next, across different prompting strategies, bug categories, and quantum programs. Our results show that iterative feedback is critical, as a single retry raises Pass@1 from below 25% to above 80%. Moreover, simpler structured prompting can even outperform Chain-of-Thought and ReAct for reasoning-capable models under fixed-resource constraints. Our work takes initial steps toward benchmarking LLM capabilities for debugging quantum programs and offers practical insights to support future efforts in automated quantum software repair.
Sources
- Qiskit Code Assistant: Training LLMs for generating Quantum Computing Code
- Qiskit HumanEval: An Evaluation Benchmark For Quantum Code Generative Models
- Leveraging Mutation Analysis for LLM-based Repair of Quantum Programs
- Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- ReAct: Synergizing Reasoning and Acting in Language Models
- Evaluating Large Language Models Trained on Code
- Asking the Right Questions: Improving Reasoning with Generated Stepping Stones
Related papers
- Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits
- GitSkills: A Dataset of Agent Skills on GitHub
- SABER: Benchmarking Operational Safety of LLM Coding Agents in Stateful Project Workspaces
- PackMonitor: Enabling Zero Package Hallucinations Through Decoding-Time Monitoring
- IntentCoding: Amplifying User Intent in Code Generation
- Incentives and Outcomes in Bug Bounties