Rolling Round-Robin Rate: Standard Heterogeneous Throughput for SPEC CPU
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: We're looking at the paper: "Rolling Round-Robin Rate".
Jane: Here's the gist of the paper. SPEC CPU 2026 introduces Rolling RoundRobin Rate (RRR), a new exhibition run style for this purpose.
Tom: This paper describes RRR as a benchmark methodology and proposes a SPEC-like scoring model for future RRR reporting.
Jane: The proposed RRR reporting extension supports a practical score by averaging normalized throughput for each benchmark across copies, and by computing a SPEC-style geomean for each copy across the suite.
Tom: Benchmark Methodology 1.
Paper summary: Tom: So we’re looking at this paper today called "Rolling Round-Robin Rate: Standard Heterogeneous Throughput for SPEC CPU." It seems to be introducing a new way to test how processors handle mixed workloads, not just running everything at once.
Jane: Right, so what does this paper claim is the main idea? What’s the big picture they’re trying to get across about this Rolling Round-Robin Rate method?
Tom: Basically, they take the existing SPEC rate benchmarks and use a deterministic rolling schedule to create these mixed workloads. It’s about setting up a specific kind of heterogeneous test that mimics real-world server or cloud scenarios where things are running at different times and interacting with each other.
Lu: The core methodology they describe is that every copy runs all the selected benchmarks in sequence, but they use the same fixed order of benchmarks for every copy, just starting at a different spot in that sequence. This creates these deterministic heterogeneous loads because if you change where you start on one copy, it’s different from another.
Meng: From my side at the production level, that sounds like something we need to worry about when we talk about how a system will actually behave under load. How does this deterministic scheduling help us understand what breaks?
Lalam: It focuses on the CPU behavior specifically, as they select and harden the SPEC CPU benchmarks to reduce things like file I/O or unstable random behavior, which is good because it keeps the focus tight on what the processor is doing >
Jane: So if we think about a listener who isn't reading this, what's the naive question they might have? What would they ask when they hear about this Rolling Round-Robin Rate?
Tom: They’d probably ask how to get a reliable throughput number for a machine running a mix of different tasks, because usually, you just measure things one way or another.
Jane: Exactly. And what does the paper claim is the proposed scoring model for this new RRR reporting extension? What makes their new way of measuring it special?
Tom: They propose two main ways to get a practical score. First, they suggest averaging the normalized throughput for each specific benchmark across all those copies you ran. That gives you a per-benchmark average.
Lu: And second, they also say you should compute a SPEC-style geomean for each individual copy across the entire suite of benchmarks. This gives a different perspective on how balanced that single machine is performing >
Meng: Averaging across copies sounds helpful for getting away with some noise from one specific run, but what about the variability? What if one copy runs really well and another copy struggles because of something unique to its placement in the schedule?
Paper summary: Lalam: That’s where they get into the diagnostic value. They look at the coefficient of variation spread of those per-copy geomeans. This exposes things that are usually hidden, like how sensitive a specific benchmark is to which other benchmarks are running alongside it >
Jane: So if you were listening, what would be the implication for someone just trying to understand this? What does this mean for their day-to-day experience with computing?
Tom: It means we get a standardized way to look at mixed-load performance. It’s giving industry folks and academics a common experiment they can run instead of having to build custom workload mixes every single time >
Lu: For researchers, it gives them that baseline for studies where they are currently building their own specific workload mixes, which is super useful when you're trying to compare different things >
Meng: But I have to ask about the practical side. The paper says the rolling schedule doesn't keep the machine at a perfectly constant heterogeneous load throughout the entire run. That’s a caveat, right?
Tom: Yeah, that’s a key limitation. It means it doesn't guarantee perfect steady-state behavior for every single second of the test run >
Jane: And what about how you set up this rolling schedule itself? How do you control how those copies advance through the roster? What are the options there?
Lu: The paper says there’s a parameter called an increment value, or inc. Different increments give different deterministic schedules, and relatively prime increments create the most diverse roster rotation possible >
Meng: If we're talking about deployment, I worry about complexity. How hard is it to set up this experiment compared to just running a standard throughput test?
Lalam: The paper says it’s deterministic, so you can specify the suite and copy count and the increment value, which means you can reconstruct the schedule exactly if you want >
Jane: So what’s on the table when we look at this Rolling Round-Robin Rate paper in terms of its overall message? What should someone remember about it after hearing this discussion?
Tom: It extends SPEC CPU throughput measurement into a standardized heterogeneous setting. The whole point is providing a common experiment for mixed-workload CPU performance >
Lu: The core proposal is that the RRR reporting extension supports a practical score by averaging normalized throughput for each benchmark across copies and computing a SPEC-style geomean for each copy across the suite >
Meng: And remember, those headline scores aren't interchangeable. You can't just compare them directly because they stress the processor differently and answer different performance questions >
Lalam: It gives the community a common experiment for mixed-workload CPU throughput and a richer basis for comparing modern systems >
Conclusion: Tom: So we’re wrapping up this look at Rolling Round-Robin Rate, which is about testing CPU throughput under mixed workloads in a standardized way.
Jane: Yeah, so what’s the real takeaway here for someone just listening? It boils down to getting a solid number for how a processor handles different kinds of tasks at the same time.
Tom: Exactly. The authors Bares, they introduce this new reporting extension that lets you calculate two scores: one based on averaging things across copies, and another based on calculating these specific geomeans for each copy.
Lu: What they’re actually doing is setting up a deterministic rolling schedule for SPEC rate benchmarks across different copies so you get this mixed load scenario we see in the real world.
Meng: The main thing I hear is that it gives everyone, from silicon designers to cloud providers, a repeatable experiment for mixed-workload throughput. That standardization is useful when you’re trying to compare different systems or compiler teams.
Jane: It’s like they’re giving us a common yardstick for mixed-load performance instead of everyone building their own test environment from scratch every time.
Tom: Right, and they make it clear that these headline scores aren't interchangeable. You can't just compare the benchmark average score to the copy geomean score directly because they stress the processor in different ways.
Lalam: That variability is what’s interesting; it shows how much each specific task reacts to which other tasks are running alongside it and how balanced that single machine is across the whole suite.
Jane: It definitely points toward a richer way to understand those performance numbers, showing you not just the mean but also where the spread is coming from.
Tom: So, in short, RRR standardizes how we measure mixed CPU throughput by giving us these layered scores for averaging and geomeans. But there are still limitations on how perfectly steady that load stays across the whole test run.
Lu: That’s right; it provides a common baseline for studies building custom workload mixes, but you gotta remember those specific scheduling details matter when you look at the fine print.
Ampere Computing
cs.PF, cs.AR, cs.DC
Submitted: 2026-08-30
Updated: 2026-08-30
Comments: 6 pages, 1 figure, Presented at the 2026 IEEE International Symposium on Workload Characterization (IISWC 2026), Boulder, Colorado
DOI: 10.1109/IISWC71020.2026.00070
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 83/100
The gist: SPEC CPU 2026 introduces Rolling RoundRobin Rate (RRR), a new exhibition run style for this purpose.
Key concepts
- Deterministic Rolling Schedule
- RRR runs SPEC rate benchmarks in a fixed, rolling sequence. Every copy executes the same set of benchmarks but starts at different points in the sequence. This method ensures a standardized way to mix workloads across multiple CPU copies without using synchronization barriers during execution.
- Benchmark-level RRR Throughput Score
- This score is calculated for each specific benchmark and each CPU copy. It is determined by taking the arithmetic mean of the normalized throughput ratios recorded for that benchmark across all available copies, showing how well a single benchmark performs under different co-runner conditions.
- SPECrrrrate
- This is the proposed headline suite score. It is calculated by taking the arithmetic mean of the SPEC-style geomeans computed for each individual copy across the entire suite of benchmarks. This provides a comprehensive, averaged measure of overall system performance under mixed loads.
- Coefficient of Variation Spread
- This metric analyzes the spread (variability) among different copy-level geomeans. It reveals information about how sensitive specific benchmarks are to which co-runners are present and shows the relative balance or spread around the final suite score.
Terminology
Summary
SPEC CPU 2026 introduces Rolling RoundRobin Rate (RRR), a new exhibition run style for this purpose. p. 0 Researchers have repeatedly constructed their own multiprogrammed workloads to study these effects, but the field lacks
a widely adopted, standard workload construction method for
full-application heterogeneous CPU rate measurements. SPEC CPU 2026 introduces Rolling Round-Robin Rate
(RRR), a new exhibition run style for this purpose [7–9]. SPEC CPU 2026 introduces Rolling RoundRobin Rate (RRR), an exhibition run style that uses the existing
rate suites to generate deterministic heterogeneous multiprogrammed workloads. This paper describes RRR as a benchmark
methodology and proposes a SPEC-like scoring model for future
RRR reporting: compute per-benchmark average throughput
and coefficient of variation from per-copy ratios, compute a
SPEC-style geometric mean for each copy across the suite, and
average that population of copy geomeans to form a suite score
with its own coefficient of variation. The proposed RRR reporting extension also supports a
practical score. By averaging normalized throughput for each
benchmark across copies, and by computing a SPEC-style
geomean for each copy across the suite, RRR stays close
to the SPEC scoring tradition of ratios against a reference
machine. The proposed RRR reporting extension also supports a
practical score. By averaging normalized throughput for each
benchmark across copies, and by computing a SPEC-style
geomean for each copy across the suite, RRR stays close
to the SPEC scoring tradition of ratios against a reference
machine. The proposed RRR reporting extension also supports a
practical score. By averaging normalized throughput for each
benchmark across copies, and by computing a SPEC-style
geomean for each copy across the suite, RRR stays close
to the SPEC scoring tradition of ratios against a reference
machine. The proposed RRR reporting extension also supports a
practical score. By averaging normalized throughput for each
benchmark across copies, and by computing a SPEC-style
geomean for each copy across the suite, RRR stays close
to the SPEC scoring tradition of ratios against a reference
machine.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Benchmark Methodology
-
RRR runs the SPEC rate benchmarks in a deterministic rolling schedule. p. 0
p. 0Where the paper says it
SPEC CPU 2026 introduces Rolling Round-Robin Rate (RRR), a new exhibition run style for this purpose [7–9]. RRR runs the SPEC rate benchmarks in a deterministic rolling schedule: every copy runs the same selected roster, but each copy starts at a different point and then walks through the roster.
Where the paper says it
SPEC CPU 2026 introduces Rolling Round-Robin Rate (RRR), a new exhibition run style for this purpose [7–9]. RRR runs the SPEC rate benchmarks in a deterministic rolling schedule: every copy runs the same selected roster, but each copy starts at a different point and then walks through the roster.
-
Every copy runs all selected benchmarks sequentially, using the same deterministic roster but beginning at different positions. p. 1
p. 1Where the paper says it
Given a selected benchmark roster B with N benchmarks and a copy count M, each copy runs all selected benchmarks sequentially. The copies use the same deterministic roster but begin at different positions.
Where the paper says it
Given a selected benchmark roster B with N benchmarks and a copy count M, each copy runs all selected benchmarks sequentially. The copies use the same deterministic roster but begin at different positions.
-
The harness does not insert any barriers or synchronization points while the roster is executing. p. 1
p. 1Where the paper says it
After each benchmark finishes, the copy advances through the roster until every copy has run every selected benchmark once. All verification is delayed until the end of all N × M benchmarks have completed; the harness does not insert any barriers or synchronization points while the roster is executing.
Where the paper says it
After each benchmark finishes, the copy advances through the roster until every copy has run every selected benchmark once. All verification is delayed until the end of all N × M benchmarks have completed; the harness does not insert any barriers or synchronization points while the roster is executing.
-
The CPU 2026 harness exposes this mode through-rrrrate; the-rrrrate inc option controls how each copy advances through the roster. p. 1
p. 1Where the paper says it
All verification is delayed until the end of all N × M benchmarks have completed; the harness does not insert any barriers or synchronization points while the roster is executing. The CPU 2026 harness exposes this mode through -rrrrate; the -rrrrate_inc option controls how each
Where the paper says it
All verification is delayed until the end of all N × M benchmarks have completed; the harness does not insert any barriers or synchronization points while the roster is executing. The CPU 2026 harness exposes this mode through -rrrrate; the -rrrrate_inc option controls how each
Scoring Model Extension
The proposed RRR reporting extension supports a practical score by averaging normalized throughput for each benchmark across copies, and by computing a SPEC-style geomean for each copy across the suite.
-
For each benchmark and copy, define xb,c as the copy’s representative ratio. p. 2
p. 2Where the paper says it
The normalized rate ratio for one execution is xb,c,i = M · Rb Tb,c,i . (1) For each benchmark and copy, define xb,c as the copy’s representative ratio.
Where the paper says it
The normalized rate ratio for one execution is xb,c,i = M · Rb Tb,c,i . (1) For each benchmark and copy, define xb,c as the copy’s representative ratio.
-
The benchmark-level RRR throughput score is the arithmetic mean of those per-copy ratios. p. 2
p. 2Where the paper says it
With one iteration this is simply xb,c,1; with multiple iterations it follows the usual SPEC practice of selecting the median of three runs for that copy. The benchmark-level RRR throughput score is the arithmetic mean of those per-copy ratios: ratio_avgb = ¯xb = 1 M X M c=1 xb,c.
Where the paper says it
With one iteration this is simply xb,c,1; with multiple iterations it follows the usual SPEC practice of selecting the median of three runs for that copy. The benchmark-level RRR throughput score is the arithmetic mean of those per-copy ratios: ratio_avgb = ¯xb = 1 M X M c=1 xb,c.
-
The proposed headline suite score, SPECrrrrate, is the arithmetic mean of the per-copy geomeans. p. 2
p. 2Where the paper says it
For each copy, compute a SPEC-style geometric mean across the benchmark ratios observed by that copy: Gc = Y b∈B xb,c! (4) The proposed headline suite score, SPECrrrrate, is the arithmetic mean of the per-copy geomeans: basemean_avg = SRRR = 1 M X M c=1 Gc.
Where the paper says it
For each copy, compute a SPEC-style geometric mean across the benchmark ratios observed by that copy: Gc = Y b∈B xb,c! (4) The proposed headline suite score, SPECrrrrate, is the arithmetic mean of the per-copy geomeans: basemean_avg = SRRR = 1 M X M c=1 Gc.
Diagnostic Value and Interpretation
The coefficient of variation spread of those geomeans exposes information that homogeneous rate largely hides: benchmark sensitivity to co-runners, copy-level balance, and relative spread around the final score.
-
At the benchmark level, ratio avg and ratio cv would show mean throughput and relative spread for each benchmark under heterogeneous co-runner contexts. p. 2
p. 2Where the paper says it
(6) With this extension, RRR would yield useful statistics at two levels. At the benchmark level, ratio_avg and ratio_cv would show mean throughput and relative spread for each benchmark under heterogeneous co-runner contexts.
Where the paper says it
(6) With this extension, RRR would yield useful statistics at two levels. At the benchmark level, ratio_avg and ratio_cv would show mean throughput and relative spread for each benchmark under heterogeneous co-runner contexts.
-
At the suite level, basemean avg and basemean cv would show the mean and relative spread of complete copy-level suite geomeans. p. 3
p. 3Where the paper says it
suite level, basemean_avg and basemean_cv would show the mean and relative spread of complete copy-level suite geomeans. These four quantities allow a result to answer both “how much throughput did the system deliver?
Where the paper says it
suite level, basemean_avg and basemean_cv would show the mean and relative spread of complete copy-level suite geomeans. These four quantities allow a result to answer both “how much throughput did the system deliver?
Utility in Research
RRR is useful because it standardizes a workload shape that many communities already need.
-
For industry, it provides a repeatable mixed-load throughput experiment for CPU buyers, server vendors, cloud providers, operating system teams, compiler teams, and silicon designers. p. 3
p. 3Where the paper says it
USES FOR INDUSTRY AND ACADEMIA RRR is useful because it standardizes a workload shape that many communities already need. For industry, it provides a repeatable mixed-load throughput experiment for CPU buyers, server vendors, cloud providers, operating system teams, compiler teams, and silicon designers.
Where the paper says it
USES FOR INDUSTRY AND ACADEMIA RRR is useful because it standardizes a workload shape that many communities already need. For industry, it provides a repeatable mixed-load throughput experiment for CPU buyers, server vendors, cloud providers, operating system teams, compiler teams, and silicon designers.
-
For academia, RRR offers a common baseline for studies that currently build custom workload mixes. p. 3
p. 3Where the paper says it
The same RRR run can support average-throughput comparison, perbenchmark sensitivity analysis, and variability analysis for noisy-neighbor behavior. For academia, RRR offers a common baseline for studies that currently build custom workload mixes.
Where the paper says it
The same RRR run can support average-throughput comparison, perbenchmark sensitivity analysis, and variability analysis for noisy-neighbor behavior. For academia, RRR offers a common baseline for studies that currently build custom workload mixes.
-
The run uses full SPEC applications so results remain connected to a widely studied benchmark suite whose behavior has been characterized over multiple generations. p. 3
p. 3Where the paper says it
Compiler and architecture studies can examine whether an optimization improves the average but increases relative spread, or whether it improves stability without changing the headline score. Because the run uses full SPEC applications, results remain connected to a widely studied benchmark suite whose behavior has been characterized over multiple generations [49–51].
Where the paper says it
Compiler and architecture studies can examine whether an optimization improves the average but increases relative spread, or whether it improves stability without changing the headline score. Because the run uses full SPEC applications, results remain connected to a widely studied benchmark suite whose behavior has been characterized over multiple generations [49–51].
Limitations and Context
-
The rolling schedule does not keep the machine at a perfectly constant heterogeneous load for the entire run. p. 3
p. 3Where the paper says it
LIMITATIONS RRR has limitations that should be stated explicitly. First, the rolling schedule does not keep the machine at a perfectly constant heterogeneous load for the entire run.
Where the paper says it
LIMITATIONS RRR has limitations that should be stated explicitly. First, the rolling schedule does not keep the machine at a perfectly constant heterogeneous load for the entire run.
-
RRR’s mixing depends on the increment, where relatively prime increments give the most diverse roster rotation, while increments that share a factor with N repeat alignments sooner and reduce co-runner diversity. p. 3
p. 3Where the paper says it
For a roster of N benchmarks and increment k, the modular walk visits every benchmark before repeating only when gcd(k,N)=1. Relatively prime increments therefore give the most diverse roster rotation, while increments that share a factor with N repeat alignments sooner and reduce co-runner diversity.
Where the paper says it
For a roster of N benchmarks and increment k, the modular walk visits every benchmark before repeating only when gcd(k,N)=1. Relatively prime increments therefore give the most diverse roster rotation, while increments that share a factor with N repeat alignments sooner and reduce co-runner diversity.
-
RRR’s equal-exposure property is count-based, not time-weighted, meaning longer benchmarks naturally overlap with more phases and peer benchmarks than shorter ones. p. 3
p. 3Where the paper says it
Finally, RRR’s equal-exposure property is count-based, not time-weighted. Each selected benchmark runs once on every copy, but longer-running benchmarks naturally overlap with more phases and peer benchmarks than shorter-running ones.
Where the paper says it
Finally, RRR’s equal-exposure property is count-based, not time-weighted. Each selected benchmark runs once on every copy, but longer-running benchmarks naturally overlap with more phases and peer benchmarks than shorter-running ones.
-
SPECrrrrate answers a heterogeneous throughput question and exposes the variability around that answer. p. 3
p. 3Where the paper says it
SPECrate answers a homogeneous throughput question. SPECrrrrate answers a heterogeneous throughput question and exposes the variability around that answer.
Where the paper says it
SPECrate answers a homogeneous throughput question. SPECrrrrate answers a heterogeneous throughput question and exposes the variability around that answer.
Conclusion
Rolling RoundRobin Rate extends SPEC CPU throughput measurement to a standardized heterogeneous setting. The proposed RRR reporting extension supports a practical score by averaging normalized throughput for each benchmark across copies, and by computing a SPEC-style geomean for each copy across the suite. RRR gives the community a common experiment for mixed-workload CPU throughput and a richer basis for comparing modern systems. p. 3 We expect
these diagnostics to be useful for studying noisy neighbors, OS
scheduling, heterogeneous systems, resource partitioning, and
future benchmark methodology. RRR gives the community a
common experiment for mixed-workload CPU throughput and
a richer basis for comparing modern systems. We expect
these diagnostics to be useful for studying noisy neighbors, OS
scheduling, heterogeneous systems, resource partitioning, and
future benchmark methodology. RRR gives the community a
common experiment for mixed-workload CPU throughput and
a richer basis for comparing modern systems. We expect
these diagnostics to be useful for studying noisy neighbors, OS
scheduling, heterogeneous systems, resource partitioning, and
future benchmark methodology. RRR gives the community a
common experiment for mixed-workload CPU throughput and
a richer basis for comparing modern systems. SPECrrrrate answers a heterogeneous throughput question and
exposes the variability around that answer. These headline
scores are not interchangeable and should not be compared
directly since each stresses the processor differently and answers a different performance question. SPECrrrrate answers a heterogeneous throughput question and
exposes the variability around that answer. These headline
scores are not interchangeable and should not be compared
directly since each stresses the processor differently and answers a different performance question.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
REFERENCE
Bares Rolling RoundRobin Rate extends SPEC CPU throughput measurement to a standardized heterogeneous setting. The proposed RRR reporting extension supports a practical score by averaging normalized throughput for each benchmark across copies, and by computing a SPEC-style geomean for each copy across the suite. RRR gives the community a common experiment for mixed-workload CPU throughput and a richer basis for comparing modern systems.
Bares Benchmark Methodology 1. p. 0 SPEC CPU 2026 introduces Rolling RoundRobin Rate (RRR), an exhibition run style that uses the existing
rate suites to generate deterministic heterogeneous multiprogrammed workloads. This paper describes RRR as a benchmark
methodology and proposes a SPEC-like scoring model for future
RRR reporting: compute per-benchmark average throughput
and coefficient of variation from per-copy ratios, compute a
SPEC-style geometric mean for each copy across the suite, and
average that population of copy geomeans to form a suite score
with its own coefficient of variation. SPEC CPU 2026 introduces Rolling RoundRobin Rate (RRR), an exhibition run style that uses the existing
rate suites to generate deterministic heterogeneous multiprogrammed workloads. This paper describes RRR as a benchmark
methodology and proposes a SPEC-like scoring model for future
RRR reporting: compute per-benchmark average throughput
and coefficient of variation from per-copy ratios, compute a
SPEC-style geometric mean for each copy across the suite, and
average that population of copy geomeans to form a suite score
with its own coefficient of variation.Where the paper says it
Where the paper says it
Bares Scoring Model Extension The proposed RRR reporting extension supports a practical score by averaging normalized throughput for each benchmark across copies, and by computing a SPEC-style geomean for each copy across the suite. p. 3 The proposed RRR reporting extension also supports a
practical score. By averaging normalized throughput for each
benchmark across copies, and by computing a SPEC-style
geomean for each copy across the suite, RRR stays close
to the SPEC scoring tradition of ratios against a reference
machine. The proposed RRR reporting extension also supports a
practical score. By averaging normalized throughput for each
benchmark across copies, and by computing a SPEC-style
geomean for each copy across the suite, RRR stays close
to the SPEC scoring tradition of ratios against a reference
machine.Where the paper says it
Where the paper says it
Bares Diagnostic Value and Interpretation The coefficient of variation spread of those geomeans exposes information that homogeneous rate largely hides: benchmark sensitivity to co-runners, copy-level balance, and relative spread around the final score. p. 3 By averaging normalized throughput for each
benchmark across copies, and by computing a SPEC-style
geomean for each copy across the suite, RRR stays close
to the SPEC scoring tradition of ratios against a reference
machine. Averaging the population of copy geomeans gives a
practical suite score, while the coefficient of variance spread
of those geomeans exposes information that homogeneous rate
largely hides: benchmark sensitivity to co-runners, copy-level
balance, and relative spread around the final score. By averaging normalized throughput for each
benchmark across copies, and by computing a SPEC-style
geomean for each copy across the suite, RRR stays close
to the SPEC scoring tradition of ratios against a reference
machine. Averaging the population of copy geomeans gives a
practical suite score, while the coefficient of variance spread
of those geomeans exposes information that homogeneous rate
largely hides: benchmark sensitivity to co-runners, copy-level
balance, and relative spread around the final score.Where the paper says it
Where the paper says it
Bares Utility in Research RRR is useful because it standardizes a workload shape that many communities already need. p. 3 This preserves full-application behavior, but means RRR
should be interpreted as a standardized heterogeneous throughput experiment, not a perfectly balanced pairwise co-runner
matrix. USES FOR INDUSTRY AND ACADEMIA
RRR is useful because it standardizes a workload shape that
many communities already need. This preserves full-application behavior, but means RRR
should be interpreted as a standardized heterogeneous throughput experiment, not a perfectly balanced pairwise co-runner
matrix. USES FOR INDUSTRY AND ACADEMIA
RRR is useful because it standardizes a workload shape that
many communities already need.Where the paper says it
Where the paper says it
Bares Limitations and Context 1. p. 3 On heterogeneous systems, they
may reflect core type, thermal policy, OS scheduling, or
resource partitioning. LIMITATIONS
RRR has limitations that should be stated explicitly. On heterogeneous systems, they
may reflect core type, thermal policy, OS scheduling, or
resource partitioning. LIMITATIONS
RRR has limitations that should be stated explicitly.Where the paper says it
Where the paper says it
Bares Conclusion Rolling RoundRobin Rate extends SPEC CPU throughput measurement to a standardized heterogeneous setting. p. 3 Taken together, these
modes give a more complete view of CPU performance in the
modern many-core era. CONCLUSION
Rolling Round-Robin Rate extends SPEC CPU throughput measurement to a standardized heterogeneous setting. Taken together, these
modes give a more complete view of CPU performance in the
modern many-core era. CONCLUSION
Rolling Round-Robin Rate extends SPEC CPU throughput measurement to a standardized heterogeneous setting.Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Rolling Round-Robin Rate (RRR) introduces a new exhibition run style that uses existing rate suites to generate deterministic heterogeneous multiprogrammed workloads, which allows for controlled heterogeneous load testing in modern cloud and server systems.
-
The proposed SPEClike RRR scoring model computes
compute per-benchmark average throughput and coefficient of variation from per-copy ratios,
creating a benchmark methodology that preserves the SPEC throughput tradition while exposing richer information about variability, interference, and heterogeneous system behavior. -
The suite score is formed by averaging the population of copy geomeans, which allows the resulting score to answer both “how much throughput did the system deliver?” and “how evenly and predictably did it deliver that throughput under heterogeneous load?
Abstract
SPEC CPU has long provided a common foundation for comparing processor, compiler, memory-system, and platform performance. Its multi-copy SPECrate mode measures homogeneous throughput by running many copies of the same benchmark at once. That mode remains valuable, but modern cloud and server systems commonly run heterogeneous collections of jobs whose interactions are shaped by shared caches, memory bandwidth, power management, operating-system scheduling, and noisy neighbors. SPEC CPU 2026 introduces Rolling Round-Robin Rate (RRR), an exhibition run style that uses the existing rate suites to generate deterministic heterogeneous multiprogrammed workloads. This paper describes RRR as a benchmark methodology and proposes a SPEC-like scoring model for future RRR reporting: compute per-benchmark average throughput and coefficient of variation from per-copy ratios, compute a SPEC-style geometric mean for each copy across the suite, and average that population of copy geomeans to form a suite score with its own coefficient of variation. RRR therefore preserves the familiar SPEC throughput tradition while exposing richer information about variability, interference, and heterogeneous-system behavior.
Sources
Related papers
- Instruction-Aware Transfer of GPU Implementation Crossovers: a Tensor-Product Discontinuous Galerkin Operator on GB200, H100, and A100
- denet, A lightweight command-line tool for process monitoring in benchmarking and beyond
- HybridQC: Hardware-Grounded Simulation of Tightly Integrated Hybrid Quantum-Classical Systems
- TileBench: A Controlled Benchmark for Performance Evaluation and Bottleneck Diagnosis of Tile-Based Programming Models
- The M/PH 2/1 Queue: Explicit Distributions, Sojourn Times and Numerical Benchmarks
- Reproducible LLM Inference Benchmarking: A Sequential Isolation Protocol for Regression Testing