The Cost of Synchronization in GPU Scale-Up Domains
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: We're looking at the paper: "The Cost of Synchronization in GPU Scale-Up Domains".
Title and authors: Tom: So we're looking at this paper called "The Cost of Synchronization in GPU Scale-Up Domains". The title itself tells you right away that we're talking about how communication between GPUs gets slower when you make the domain bigger.
Jane: Right, and the authors are Arjun Devraj Lindsey Bowen Rachee Singh from Cornell University. It’s interesting because they’re looking at something that happens when we try to build really big systems for machine learning.
Lu: It's about how GPUs within a scale-up domain arrive at collective barriers hundreds to thousands of microseconds apart, even when they're all running the same code on the same hardware over a uniform fabric. That’s a huge gap between the design goal and what we see in practice, so it sets up this whole problem.
Meng: I mean, if you're trying to train something massive, that waiting time sounds like wasted compute time just sitting idle. We need to know if this is a hardware limitation or something about how we’re organizing the communication.
Lalam: The authors are showing that when you have an eight-GPU scale-up domain, the synchronization tax can eat up over fifty percent of the total collective communication time, which is a pretty big number for training efficiency <ref:2608.22503#pg1,an 8-GPU scale-up domain>.
Tom: Fifty percent is significant; it means we're not getting the full benefit of our interconnect bandwidth when we try to scale up these clusters. It’s about more than just slow communication; it’s about how much of the time the GPUs are actually doing useful work versus waiting around for a signal.
Jane: So, what they do next is look into what causes this tax and how big it gets as you add more GPUs to that scale-up domain. It moves from describing the problem to figuring out where the problem is coming from.
The paper's summary: Tom: The core idea of this paper, "The Cost of Synchronization in GPU Scale-Up Domains", is that we have two things growing exponentially: interconnect bandwidth and the size of the scale-up domain. These two trends are in tension because they create this synchronization tax.
Jane: So, to put it plainly, as you add more GPUs to a machine learning setup, the time it takes for them all to meet at a collective barrier doesn't stay constant; it grows much faster than we expect.
Lu: They study several hundred thousand collective operations across four different language models and three recent GPU architectures. They found that the identity of the slowest rank changes every time you run a different collective operation, which means it’s not just one bad piece of hardware causing the delay; it’s something systemic in how things are scheduled.
Meng: That's worrying because if it was just a single faulty chip, we could fix that with replacement. If the problem is in the scheduling or the kernel execution variation, then fixing it requires changing how we run things at a fundamental level.
Lalam: They design a graph-based algorithm that looks at per-rank kernel traces to figure out where this overhead comes from, and they found that cross-rank variation in GEMM kernel execution times is responsible for seventy eight percent of this overhead.
Tom: Seventy eight percent of the delay coming from how one GPU handles matrix multiplications differently than another, even when they are on the same hardware and fabric. That's a lot of computational noise hidden inside our training time.
Jane: So, what they show is that this synchronization delay isn't just some random hiccup; it’s tied to the way different parts of the computation interact across those GPUs, especially in tensor parallelism scenarios like tensor parallel parallelism and fully sharded data parallelism.
The paper's improvements: Tom: Now, the authors don't just stop there; they suggest ways to model this variation using extreme value theory. They use that theory to show that this synchronization tax actually grows as the domain size gets bigger. That’s a key finding because it links the problem directly to scaling behavior.
Jane: So, what this means for us is that the synchronization overhead isn't just a fixed cost; it gets worse when we try to run things on much larger GPU clusters. It effectively taxes the benefits of adding more bandwidth.
Lu: They build on that by proposing a bandwidth scaling model. This model shows that as the scale-up domain size grows, these synchronization overheads increase, which means you need less physical interconnect bandwidth than the simple baseline models predicted for huge systems.
Meng: That’s an important implication for deployment. If we design our infrastructure based on older assumptions about how much bandwidth we need, we might be over-provisioning or under-provisioning in a way that doesn't actually match what the actual training time looks like.
Lalam: They find that for a five hundred twelve GPU scale-up domain, you only need two point zero six times less optimal bandwidth compared to an eight GPU domain, which is a concrete number for planning purposes.
Tom: So we’re looking at needing lower interconnect bandwidth requirements as the cluster gets larger, because the synchronization tax eats up that extra potential bandwidth. That’s a really counterintuitive result if you're used to thinking more GPUs mean more bandwidth needed.
Conclusion: Jane: So, wrapping up this paper, "The Cost of Synchronization in GPU Scale-Up Domains" shows us that the synchronization tax is real and it scales with domain size. It’s driven primarily by how much the execution time of matrix multiplication kernels varies between different GPUs.
Tom: The main practical implication for me is that when we design our next generation systems, we can’t just focus on raw interconnect speed anymore; we have to factor in this synchronization tax into our communication budget.
Lu: For future work, they're looking at incorporating their findings into a more complete communication cost model using the Hockney model. This would allow us to calculate the optimal bandwidth requirement by explicitly including that synchronization tax term in the formula for B star.
Meng: From an engineering standpoint, if we can accurately predict this delay, it helps us decide what to deploy and how much hardware to budget for when we move into those massive scale-up domains.
Lalam: I'm confident that by incorporating this synchronization tax into the model, we can get a much more precise prediction of the required bandwidth for any given training setup.
Tom: So, in short, this paper is telling us that as GPU clusters grow larger, the communication bottleneck isn't just about the wires; it’s about waiting around for each other to finish their local work. That's what we learned from "The Cost of Synchronization in GPU Scale-Up Domains".
Cornell University
cs.DC
Submitted: 2026-08-23
Updated: 2026-11-18
Code: https://github.com/NVIDIA/nccl-tests
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 83/100
The gist: GPU scale-up domains have become the building block of modern machine learning infrastructure, and their design follows a clear trajectory of exponential growth in both interconnect bandwidth and…
Key concepts
- Synchronization Tax
- This is the idle time GPUs spend waiting for the slowest rank to finish a collective operation before proceeding. It occurs because ranks must wait for the maximum execution time across all devices, not just the fastest ones.
- Decomposition of Overhead
- The total communication time is split into two parts: a 'wait phase' where GPUs are idle, and a 'transfer phase' where data moves. The paper found that 78% of this waiting overhead comes from differences in how individual GPUs execute matrix multiplication kernels (GEMM).
Terminology
Summary
GPU scale-up domains have become the building block of modern machine learning infrastructure, and their design follows a clear trajectory of exponential growth in both interconnect bandwidth and domain size <ref:2608. p. 0 Understanding the Synchronization Tax in GPU Scale-Up Domains
Arjun Devraj Lindsey Bowen Rachee Singh
Cornell University
Abstract
GPU scale-up domains have become the building block of
modern machine learning infrastructure, and their design follows a clear trajectory of exponential growth in both interconnect bandwidth and domain size. This paper argues that these
two trends are in tension. This paper argues that these
two trends are in tension. Through a study of several hundred
thousand collective operations across four language models
and three recent GPU architectures, we find that GPUs within
a scale-up domain arrive at collective barriers hundreds to
thousands of microseconds apart, despite executing identical
kernels on identical hardware over a uniform fabric. Across every configuration, GPUs in
the scale-up domain arrive at barriers hundreds to thousands
of microseconds apart, even though they execute identical kernels on identical hardware over a uniform fabric. In an 8-GPU
1
arXiv:2608. Through a study of several hundred
thousand collective operations across four language models
and three recent GPU architectures, we find that GPUs within
a scale-up domain arrive at collective barriers hundreds to
thousands of microseconds apart, despite executing identical
kernels on identical hardware over a uniform fabric. We call
this waiting time the synchronization tax and show that it can
consume over 50% of collective communication time in an
8-GPU scale-up domain.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
The Nature of the Synchronization Tax
The synchronization tax is defined as a barrier-induced idle time that is inherent to collective communication, where ranks that reach the barrier early must wait until the slowest rank arrives <ref:2608.22503#pg2>. This waiting time is not caused by a single faulty device, as the identity of the slowest rank shifts from one collective to the next <ref:2608. p. 1 For Llama-3 70B on the DGX H200, the fastest
rank spends a median of 40%, and at worst nearly all, of its TP
communication time waiting before data transfer begins. The
identity of the slowest rank shifts from one collective to the
next, ruling out a single faulty device as the cause [26,35,73]. However, this view neglects a critical dimension
of modern scaling trends: the size of the scale-up domain. Since synchronization is governed by the maximum execution
time across GPUs, we use extreme value theory (EVT) from
statistics to model tail behavior, showing that the synchronization tax grows with GPU cluster size.Where the paper says it
Where the paper says it
Decomposition of the Overhead
The wall-clock time of a collective decomposes into two distinct phases: a wait phase where GPUs idle while waiting for the slowest rank, and a transfer phase where data exchange proceeds <ref:2608.22503#pg2>. The synchronization delay is defined as τi = t start i(l) − min j∈1…n t start j(l), where τ denotes the expected (i. p. 2 ,n} as
τi = t
start
i
(l) − min
j∈{1,. n}
t
start
j
(l), (1)
where t
start
i
(l) denotes the time when rank i enters collective l. Unless otherwise specified, we use τ to denote the expected
(i. , mean) synchronization delay experienced across ranks. We call
this waiting time the synchronization tax and show that it can
consume over 50% of collective communication time in an
8-GPU scale-up domain. To understand the sources of this tax,
we design a graph-based algorithm that operates on per-rank
kernel traces, revealing that cross-rank variation in GEMM
kernel execution times accounts for 78% of this overhead.Where the paper says it
Where the paper says it
Where the paper says it
Modeling the Growth and Scaling
Extreme value theory (EVT) is applied to model this variation, demonstrating that the synchronization tax grows with domain size <ref:2608.22503#pg2>. Building on this, a bandwidth scaling model shows that as the size of the scale-up domain grows, synchronization overheads increase, effectively taxing the benefits of additional bandwidth <ref:2608. p. 1 Building on this, we develop a bandwidth scaling model
that explicitly accounts for the synchronization tax. Our model
shows that as the size of the scale-up domain grows, synchronization overheads increase (as shown by EVT), effectively
taxing the benefits of additional bandwidth (Figure 1, right). Our model
shows that as the size of the scale-up domain grows, synchronization overheads increase (as shown by EVT), effectively
taxing the benefits of additional bandwidth (Figure 1, right). Applying our model, we find that interconnect bandwidth
requirements decrease with scale-up domain size, yielding
2. Applying our model, we find that interconnect bandwidth
requirements decrease with scale-up domain size, yielding
2. 06× lower optimal bandwidth necessary for a 512-GPU
vs.Where the paper says it
Where the paper says it
Where the paper says it
Optimal Bandwidth Requirements
The optimal bandwidth B∗ is derived by incorporating the synchronization tax τ into the classical Hockney model, resulting in B∗ = -(1+εB)· q · S/εB[p ·α+τ] <ref:2608.22503#pg2>. Because q is asymptotic in n and the synchronization tax scales logarithmically at best, B∗ decreases as n increases <ref:2608. p. 7 (3)
Max-stability allows us to predict the distribution of the maximum across larger numbers of ranks without requiring additional measurements, enabling us to estimate the synchronization tax at larger cluster sizes. 3 indicates that at best, the
synchronization tax grows logarithmically with the number
8 This trend worsens with hardware generation. For the largest model (Llama-3 70B) and latest GPU architecture evaluated (H200), the fastest rank spends a median of
40% and, at worst, nearly all of its REDUCESCATTER time
waiting to begin data transfer.Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Synchronization tax identification: An improved system can identify which preceding operations drive synchronization delays by designing a graph-based algorithm that identifies and traces through the critical path before each collective to attribute observed synchronization delay to specific cross-rank variations in GEMM kernel execution times, revealing that
cross-rank variation in GEMM kernel execution times accounts for 78% of this overhead.
-
Bandwidth scaling optimization: The system can calculate the optimal interconnect bandwidth requirement using an augmented Hockney communication cost model that explicitly accounts for the synchronization tax, yielding a formula B star which shows that
the optimal bandwidth decreases as the synchronization tax increases,
allowing for more precise provisioning decisions in growing scale-up domains. -
Predictive performance modeling: An improved system can predict the exact mean synchronization delay at any collective by fitting a General Extreme Value distribution to z-scores of cross-rank compute block execution times, enabling it to
accurately predict the mean observed synchronization delay during the collective
across various workloads and hardware architectures.
Abstract
GPU scale-up domains are the building blocks of machine learning infrastructure. Architects provision the size and bandwidth in these domains assuming all GPUs reach a collective operation at the same time. However, we find that this assumption does not hold in practice. Across 300,000 collectives from training four language models on A100, H100, and H200 domains, GPUs reach the same collective hundreds to thousands of microseconds apart, despite running identical kernels. Since a collective cannot complete until every GPU contributes its data, GPUs that arrive early wait for the slowest one. We call this waiting time the synchronization tax, and it consumes nearly 40% of tensor-parallel communication time in an 8-GPU domain. To find its source, we develop an algorithm that recovers the chain of operations that delays each GPU before a collective. Our algorithm attributes 78% of the tax to variation in the runtimes of matrix-multiplication (GEMM) kernels across GPUs. Since the tax depends on the slowest of many GPUs, our analytical model shows that it grows at least logarithmically with domain size. Thus, a 72-GPU domain requires 3.3x more per-GPU bandwidth than today's simulators predict to keep communication below 12% of step time, and the optimal scale-up fabric should partition a switch's capacity among fewer GPUs with more bandwidth each.
Sources
- Efficient AllReduce with Stragglers
- The Llama 3 Herd of Models
- EROICA: Online Performance Troubleshooting for Large-scale Model Training
- Demystifying NCCL: An In-depth Analysis of GPU Communication Protocols and Algorithms
- Lit Silicon: A Case Where Thermal Imbalance Couples Concurrent Execution in Multiple GPUs
- TrainMover: An Interruption-Resilient Runtime for ML Training
- DeepSeek-V3 Technical Report
- MSCCL++: Rethinking GPU Communication Abstractions for AI Inference
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
- Llama 2: Open Foundation and Fine-Tuned Chat Models
- FALCON: Pinpointing and Mitigating Stragglers for Large-Scale Hybrid-Parallel Training
- Qwen3 Technical Report
- Serving Large Language Models on Huawei CloudMatrix384
Related papers
- Programmable Probabilistic Computer with 1,000,000 p-bits
- BatchGen: An Architecture for Scalable and Efficient Batch Inference
- Seamless Reconfiguration for DAG-BFT (Technical Report)
- ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving
- iScheduler: Reinforcement Learning-Driven Continual Optimization for Large-Scale Resource Investment Problems
- TIRx: A Unified Tile Primitive ML Compiler Across Heterogeneous Backends