Benchmarking Quantum Computers via Protocols, Comparing Superconducting and Ion-Trap Quantum Technology

arXiv:2603.27397 · quant-ph, cs.DC, cs.ET · Submitted 2026-03-28 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Quantum Radio. Generated commentary on the latest quantum physics and condensed matter papers.

Kai: We're looking at the paper: "Benchmarking Quantum Computers via Protocols, Comparing Superconducting and Ion-Trap Quantum Technology".

Mira: Here's the gist of the paper. Both Superconducting and Ion-Trap are leading quantum architectures common in the current landscape of the quantum computing field, each with distinct characteristics and operational constraints.

Kai: Understanding and measuring the underlying quantumness of these devices is essential for assessing their readiness for practical applications and guiding future progress and research.

Mira: This work applies a novel benchmarking method based on quantum communication and state transfer primitives to compare AQT’s IBEX Q1 with IBM’s QC.

Kai: The core of the methodology involves a binary assessment of “quantumness” defined by fidelity thresholds derived from classical limits for state transfer and degraded shared fully entangled states.

Title and authors: Kai: So, we're diving into this paper called "Benchmarking Quantum Computers via Protocols, Comparing Superconducting and Ion-Trap Quantum Technology". It looks like the authors are taking two major types of quantum hardware and trying to compare them head-to-head.

Mira: Exactly. The whole point is that you can't really tell if a quantum computer is actually good unless you test it against some real physics limits, and this paper sets up a way to measure "quantumness" by looking at how well they can transfer states or share entangled states.

Lev: It seems they are using fidelity thresholds derived from the classical limits for state transfer and degraded shared fully entangled states to define what counts as quantum advantage on these different chips.

Kai: Right, so instead of just looking at some abstract metric, they're using specific protocols like Transmit, Do-nothing, and Bell-state transfer to see how each architecture performs.

Mira: They've adapted these protocols for the different systems too; they even define a "Generalized Transmit" protocol to make sure the comparison is fair between ion traps and superconducting systems.

Lev: And the fidelity thresholds they set are specific: two over three for transmit and do-nothing protocols, and half for Bell-state transfer <ref:2603.27397#pg2>. For their generalized versions, they use a threshold of two over three times the number of qubits in the work state.

Kai: That's a concrete way to measure performance, right? It moves away from just saying one chip is "better" in some abstract way and gives us actual numbers on what it can do.

Mira: And then they move into identifying an "Optimal SubChip," which they define as the largest contiguous area where there's a consistent quantumness advantage across all those tested protocols.

Lev: For IBM's processors, that optimal sub-chip is defined as the biggest set of neighboring twelve qubit rectangles that pass every protocol, or if no unified region exists, it takes the maximal set of rectangles optimal for any single protocol.

Kai: That sounds like they’re trying to find the most reliable chunk of hardware where you can actually trust the quantum performance.

Mira: Conversely, for ion-trap architectures, that optimal sub-chip is defined as a maximal set of individual qubits showing advantage for any single protocol, which reflects their unique physical layout.

Lev: The workflow they use to find this is adapted because the standard method doesn't work with AQT's chip; it involves running basic protocols first, then their generalized versions to generate a "Protocol Vector."

Kai: For IBM, the workflow is more sequential: they do the c2c stage of transmit and do-nothing first, and only then do they assess full transmit and do-nothing protocols on those successful sub-chips.

Mira: It’s an attempt to filter out the less promising parts of a chip before spending time on more complex measurements like generalized protocols.

Lev: The comparison results show that AQT’s IBEX Q1 does seem to excel at regular protocols because it has similar performance and flexible connectivity, but the Heron r2 superconducting system shows better performance on most of their generalized protocols.

Kai: So, despite the ion trap's strength in specific areas, the superconducting system seems to have a more robust general capability when you test them with those broader generalized tests.

Mira: That comparison is interesting because it highlights how architecture dictates where you get your best results across different types of quantum tasks.

Lev: The limitation they mention is that budget constraints forced them to use reduced sampling resolution for some of the ion-trap experiments, which affects the accuracy of their measurements there.

Kai: They also had a logistical challenge with temporal drift in the AQT system, meaning they had to run their entire "Optimal Lookup Workflow" in one continuous session just to keep things valid.

Mira: So, while the methodology is rigorous and comparative, it’s clear that real-world implementation has its own set of headaches when you're trying to get a definitive answer.

Lev: Looking ahead, they suggest expanding this framework to other platforms and exploring protocol-level abstraction for things like measurement based quantum computation.

Kai: And the natural evolution of the framework, as they see it, is looking at larger integrated regions, specifically successful pairs of neighboring optimal rectangles as unified sub-chips.

Mira: It sounds like they're moving from finding individual best spots to finding bigger areas that work together seamlessly.

Lev: To wrap up this discussion on "Benchmarking Quantum Computers via Protocols, Comparing Superconducting and Ion-Trap Quantum Technology," the core implication is that we have a standardized, protocol-based way to measure the actual quantum performance of different hardware platforms.

Kai: It gives us a common language to talk about quantumness beyond just some marketing claims.

Mira: It tells us exactly what kind of physical qubits are actually working well for which types of quantum operations on each architecture.

Lev: And for anyone building real hardware, it shows you exactly what overhead you're dealing with when trying to run these algorithms on the actual chips.

Kai: So, the next step is taking this framework and applying it to even more varied quantum systems and tasks.

Mira: We'll keep an eye out for papers that expand on these protocol comparisons because understanding these boundaries is crucial for where the field goes next.

Lev: That's all we have time for today on this paper, but keep an eye out for those next ones coming your way.

The paper's summary: Kai: So, this paper boils down to taking two major quantum hardware types—Superconducting and Ion-Trap—and running them through a series of specific tests to see which one is actually performing better in a real sense.

Mira: Exactly. They use these protocols, like sending states or trying to create entanglement, as a way to measure the actual "quantumness" of each system against what we know classical physics allows.

Lev: The core idea is they set up some fidelity thresholds—like two over three for simple tasks—and see which chip stays above that line across all those tests.

Kai: They found that while the Ion-Trap system does pretty well on those standard, regular protocols because it's so flexible with its connectivity, the Superconducting system actually takes the lead on most of the more complex, generalized tests they ran.

Mira: That’s interesting because it shows that what makes a chip "good" isn't just one specific task; it depends entirely on which type of operation you’re trying to do.

Lev: It also gives them a way to define an optimal subchip, which is like finding the largest connected chunk of hardware where you can actually trust the quantum performance across all those tests.

Kai: For IBM's processors, that optimal chunk was defined by looking for these neighboring twelve qubit rectangles that passed everything, and if none fit together nicely, they looked for the best single rectangle for each test.

Mira: And for AQT’s ion-trap system, they defined it differently—as the largest group of individual qubits showing advantage because their physical layout is so unique.

Lev: The workflow itself had to be adjusted because the way you run tests on that specific ion-trap chip isn't as straightforward as it is on a standard processor. They had to run basic protocols first, then their generalized versions, which they called a "Protocol Vector."

Kai: So, what this really means for us listening right now is that we have a new shared language. We can start comparing different quantum hardware architectures using these exact same benchmarks instead of just looking at marketing specs.

Mira: It shifts the focus from just saying one chip is faster to understanding *why* it's performing better in specific situations, which is crucial for designing better future systems.

Lev: From my side, this helps us understand the overhead. If you want to run a complex algorithm on a real ion-trap chip, you know exactly what kind of noise and error correction overhead you’re dealing with based on these numbers.

Kai: And the limitations they pointed out matter too—budget issues meant they had to use less precise measurements for some ion-trap tests, and temporal drift in their system forced them to run everything in one long session just to keep the data valid.

Mira: It’s a lot of work getting these rigorous results, but it sets a solid foundation for what comes next. They suggest that the next step is finding larger regions—like successful pairs of neighboring optimal rectangles—to see if those can be treated as unified subchips.

Lev: That makes sense; moving from individual best spots to bigger working areas gives you a much better picture of the system's real computational size and reliability.

Kai: So, we're not just looking at isolated components anymore; we’re starting to look at how these pieces fit together in terms of quantum performance.

Mira: That’s right. This paper is basically building a toolkit for everyone trying to figure out which hardware platform is actually ready for serious application right now.

The paper's improvements: Tom: So, we just looked at how this paper benchmarks superconducting versus ion-trap computers using state transfer protocols, and now we're looking at what they say should come next.

Kai: They suggest expanding this whole framework beyond just these two specific architectures to other quantum platforms as well.

Mira: That’s right, the authors are looking to take this same protocol-based comparison method and apply it to different types of quantum computers, which is a big step for creating a truly universal standard.

Lev: They also mentioned exploring protocol level abstraction for things that aren't just standard circuits, like measurement based quantum computation.

Kai: That means they want to move past just testing how well the chips do known tasks and start testing how well they handle entirely different kinds of quantum operations.

Mira: Exactly. The idea is to create a common language where we don't have to re-invent the testing method every time we look at a new system, whether it’s superconducting or something else entirely.

Lev: If you can abstract the protocol level, it means you can analyze the fundamental physics of how information moves around in that hardware, which is essential for developing better error correction techniques for those systems.

Kai: It also points toward a natural evolution of this benchmarking itself; they want to move from finding individual optimal regions to finding larger integrated areas, like successful pairs of neighboring rectangles as unified subchips.

Mira: That’s smart because it moves the goal from just finding a single good spot to understanding how those good spots connect and cooperate across the entire chip design.

Lev: From an engineering standpoint, that’s where you get a more realistic measure of the system's effective computational size and its overall reliability, not just one isolated measurement.

Kai: It sounds like they’re building a roadmap for how to systematically evaluate future quantum hardware as it gets more complex and integrated.

Mira: And this framework is really about making sure that when we move to bigger, more integrated systems, we have the tools ready to rigorously test their performance against those hard physical limits.

Lev: So, the implication is that this isn't just a study of AQT and IBM; it’s a proposal for how we should conduct hardware validation moving forward.

Conclusion: Kai: So we’ve gone through how this paper sets up these protocol tests to compare superconducting and ion-trap hardware, focusing on defining what actual quantum performance looks like on each system.

Mira: Right, essentially they’re giving us a standardized way to judge quantumness across different platforms using the same set of tests.

Lev: And it highlights that the results depend heavily on which physical assumptions you make about the hardware setup, so you can't just take a number out of context.

Kai: The main thing they show is that while ion-trap has strengths in regular tasks, superconducting systems tend to win on those more generalized tests they ran.

Mira: That comparison reveals how architecture really shapes where you get the best results across different types of quantum operations.

Lev: From my side, this gives us a clear picture of where the real overhead lands when we try to run these benchmarks on actual hardware like IBM’s or AQT’s systems.

Kai: It also points out some practical hurdles they hit, like needing very long continuous sessions to keep their system valid and budget cuts forcing them to use less precise measurements on the ion-trap side.

Mira: Those limitations are important because they show us that even with a good methodology, real world constraints affect how deep we can actually push the research.

Lev: Exactly, so when we think about building real error correction for these systems, we need to factor in those kinds of measurement stability issues right from the start.

Kai: So this paper on "Benchmarking Quantum Computers via Protocols, Comparing Superconducting and Ion-Trap Quantum Technology" gives us a solid comparison point.

Mira: It’s a very useful tool for anyone trying to figure out which hardware is actually ready for practical use in the near term.

Lev: It shows you exactly what kind of physical qubits are performing well under different constraints, which is crucial information for hardware developers.

Nitay Mayo, Tal Mor, Yossi Weinstein

Technion University · Helen Diller Quantum Center

quant-ph, cs.DC, cs.ET

Submitted: 2026-03-28

Updated: 2026-11-07

Comments: 19 pages, supplemental file with additional data is attached

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 82/100

The gist: Both Superconducting and Ion-Trap are leading quantum architectures common in the current landscape of the quantum computing field, each with distinct characteristics and operational constraints.

Key concepts

Quantumness Assessment
A method to measure how 'quantum' a device is by testing its ability to perform specific quantum tasks. This is done by setting fidelity thresholds based on what classical physics allows for state transfer and degraded entanglement. A higher fidelity score indicates stronger quantum performance.
Optimal Sub-Chip Identification
Identifying the largest, most reliable section of a quantum chip that consistently shows a performance advantage across various tests. For IBM, this is defined by neighboring 12-qubit rectangles; for ion traps, it focuses on individual qubits. This pinpoints the physical area where quantumness is most robust.
Protocol Vector
A characterization of a quantum computer's capabilities derived from running specific sets of tests. For AQT, this involves executing basic and generalized protocols to map out its functional strengths. This vector helps define what tasks the hardware can reliably perform.
Generalized Protocols
Modified versions of standard quantum communication tests (like Transmit or Do-nothing) adjusted for architectural differences. These versions use a parameter 'M' related to the number of qubits in the work state, allowing for a fair comparison between superconducting and ion-trap systems despite their structural variations.

Terminology

Summary

Both Superconducting and Ion-Trap are leading quantum architectures common in the current landscape of the quantum computing field, each with distinct characteristics and operational constraints. p. 0 Understanding and measuring the underlying quantumness of these devices is essential for assessing their readiness for practical applications and guiding future progress and research. p. 0 This work applies a novel benchmarking method based on quantum communication and state transfer primitives to compare AQT’s IBEX Q1 with IBM’s QC. p. 2 The core of the methodology involves a binary assessment of “quantumness” defined by fidelity thresholds derived from classical limits for state transfer and degraded shared fully entangled states. p. 2

Benchmarking Protocols

The study employs three primary protocols—Transmit, Do-nothing, and Bell-state transfer—along with their generalized versions to bridge the gap between ion-trap and superconducting architectures. Architectural differences necessitate specific adjustments to these protocols, such as defining the “Generalized Transmit” protocol for a fair comparison. p. 4 The fidelity thresholds are established as 2/3 for transmit and do-nothing protocols, and 0. p. 55 for Bell-state transfer. p. 1 For the generalized versions, a threshold of (2/3)M is set, where M represents the number of qubits in the work state. p. 5

Optimal Sub-Chip Identification

The primary output of this benchmarking methodology is the characterization of an “Optimal SubChip,” defined as the largest contiguous sub-region demonstrating a consistent quantumness advantage across tested protocols. For IBM’s processors, the optimal sub-chip is defined as the maximal set of neighboring 12-qubit rectangles that successfully pass all protocols, or the maximal set of rectangles optimal for any individual protocol if no unified region exists. p. 6 Conversely, for ion-trap architectures, the optimal sub-chip is defined as the maximal set of individual qubits demonstrating quantum advantage for any individual protocol to reflect its unique logical topology. p. 6

Workflow Adaptation

The assessment process was adapted from prior work because the standard optimal lookup workflow was incompatible with AQT’s quantum computer architecture. For AQT’s chip, the workflow involves first executing basic protocols followed by their generalized versions to generate a “Protocol Vector” characterizing its capabilities. p. 6 For IBM processors, the workflow is defined by sequential filtration steps: first c2c stage of transmit and then do-nothing, followed by full assessments of transmit and do-nothing protocols on successful sub-chips. p. 8

Comparative Performance Metrics

The comparison between the two architectures is summarized in Table 1, which presents metrics such as the “Number of qubits in the optimal reduced chip and its minimal achieved fidelity” for AQT and “The number of optimal sub-chips, along the average minimal fidelity over those sub-chips” for IBM. The analysis reveals that while AQT’s IBEX Q1 excels in regular protocols due to similar performance and flexible connectivity, the Heron-r2 superconducting system demonstrates superior performance on most generalized protocols. p. 3

Limitations and Future Directions

The research encountered logistical challenges, including budget constraints that necessitated reduced sampling resolution for ion-trap experiments. Temporal drift in the AQT system required executing the entire “Optimal Lookup Workflow” within a single continuous session to maintain validity. p. 9 Future research should focus on expanding this framework to other platforms and exploring protocol-level abstraction for non-circuit paradigms like Measurement-Based QC. p. 0 The natural evolution of the framework involves evaluating larger, integrated regions, such as successful pairs of neighboring optimal rectangles as unified sub-chips. p. 6

Acknowledgment

The authors thank AQT and IBM for offering accessible pricing models that facilitate academic research, which is essential for conducting extensive protocol-based evaluations. They also acknowledge the support from the Quantum Computing Consortium of the Israel Innovation Authority and express appreciation to specific individuals for their contributions. p. 0 The recommendation is that quantum firmware developers adopt this methodology to rigorously demonstrate quantumness and practical usability. p. 0

References

The references listed include foundational work on randomized benchmarking, volumetric frameworks, and specific protocols like teleportation and entanglement swapping. The study builds upon earlier work concerning the degradation of transferred qubits and singlet states. p. 2 Specific references for hardware architectures, such as the IBM Heron and Eagle processors, are included in Appendix A. p. 2 The work also cites studies comparing trapped-ion and superconducting architectures. p. 3 Foundational papers on quantum computation and measurement-based quantum computation are referenced in the bibliography. p. 2 The paper concludes by citing work on boson sampling and adiabatic quantum computing for future extensions of the benchmarking methodology. p. 3 The specific arXiv identifier is provided as 2603. p. 027397.

--- Page 1 ---

Benchmarking Quantum Computers via Protocols

Comparing Superconducting and Ion-Trap Quantum Technology

∗Nitay Mayo, ∗†Tal Mor and ∗Yossi Weinstein Department of Computer Science, Technion University, Haifa, Israel. †The Helen Diller Quantum Center June 3, 2026

--- Page 2 ---

Benchmarking Quantum Computers via Protocols Comparing IBM’s QC vs AQT’s IBEX Q1 Contents Contents 1 Introduction

--- Page 3 ---

Benchmarking Quantum Computers via Protocols Comparing IBM’s QC vs AQT’s IBEX Q1 Introduction

--- Page 4 ---

Benchmarking Quantum Computers via Protocols Comparing IBM’s QC vs AQT’s IBEX Q1 Introduction

--- Page 5 ---

Benchmarking Quantum Computers via Protocols Comparing IBM’s QC vs AQT’s IBEX Q1 The Protocols 2 The Protocols

--- Page 6 ---

Benchmarking Quantum Computers via Protocols Comparing IBM’s QC vs AQT’s IBEX Q1 The Protocols Figure 2

--- Page 7 ---

Benchmarking Quantum Computers via Protocols Comparing IBM’s QC vs AQT’s IBEX Q1 The Optimal Lookup Workflow 3 The Optimal Lookup Workflow

--- Page 8 ---

Benchmarking Quantum Computers via Protocols Comparing IBM’s QC vs AQT’s IBEX Q1 The Optimal Lookup Workflow

--- Page 9 ---

Benchmarking Quantum Computers via Protocols Comparing IBM’s QC vs AQT’s IBEX Q1 Introduction

--- Page 10 ---

Benchmarking Quantum Computers via Protocols Comparing IBM’s QC vs AQT’s IBEX Q1 Appendix - IBEX Q1 A.3 Bell-state transfer

--- Page 11 ---

Benchmarking Quantum Computers via Protocols Comparing IBM’s QC vs AQT’s IBEX Q1 The Optimal Lookup Workflow 4.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Benchmarking Strategy for Quantumness: The system can establish quantumness by measuring fidelity thresholds derived from classical limits of state transfer and degraded qubit-teleportation, providing a protocol-based evaluation of quantumness advantage.

  2. Optimal Sub-chip Identification: The AI can define an Optimal SubChip as the largest contiguous sub-region demonstrating consistent quantumness advantage across tested protocols, which serves as a realistic indicator of the system’s effective computational size and identifies the specific physical qubits where quantum performance is most reliable.

  3. Architecture Agnostic Comparison: The system can provide a common language for their assessment by abstracting away technical differences in the final result, allowing for fair performance comparisons between disparate quantum-circuit technologies like Superconducting and Ion-Trap architectures.

  4. Protocol Adaptability: The AI can dynamically adjust protocols based on architecture, such as defining Generalized Transmit and Generalized Do-nothing protocols with parameters M to account for differences in connectivity, noting that the swap distance depends on the path we choose in superconducting hardware versus the fixed swap distance in ion-trap systems.

  5. Workflow Optimization: The system can employ an Optimal Lookup Workflow that involves sequential filtering steps, such as executing transmit and do-nothing c2c stage first to exclude sub-chips that fail to pass a threshold before proceeding to more demanding assessments like generalized protocols.

Sources

Related papers