The Cost of Synchronization in GPU Scale-Up Domains

summary

The gist

GPU scale-up domains have become the building block of modern machine learning infrastructure, and their design follows a clear trajectory of exponential growth in both interconnect bandwidth and…

In short

The study investigated a 'synchronization tax' in large GPU clusters, finding that waiting for the slowest GPU during collective operations consumes over 50% of communication time. This tax is caused by variations in how different GPUs execute identical computations (like GEMM kernels), meaning interconnect bandwidth requirements actually decrease as cluster size grows.

Key concepts

Synchronization Tax
This is the idle time GPUs spend waiting for the slowest rank to finish a collective operation before proceeding. It occurs because ranks must wait for the maximum execution time across all devices, not just the fastest ones.
Decomposition of Overhead
The total communication time is split into two parts: a 'wait phase' where GPUs are idle, and a 'transfer phase' where data moves. The paper found that 78% of this waiting overhead comes from differences in how individual GPUs execute matrix multiplication kernels (GEMM).

Terminology used across episodes

This episode discusses

The paper

The Cost of Synchronization in GPU Scale-Up Domains · Read on arXiv

Cornell University

GPU scale-up domains are the building blocks of machine learning infrastructure. Architects provision the size and bandwidth in these domains assuming all GPUs reach a collective operation at the same time. However, we find that this assumption does not hold in practice. Across 300,000 collectives from training four language models on A100, H100, and H200 domains, GPUs reach the same collective hundreds to thousands of microseconds apart, despite running identical kernels. Since a collective cannot complete until every GPU contributes its data, GPUs that arrive early wait for the slowest one. We call this waiting time the synchronization tax, and it consumes nearly 40% of tensor-parallel communication time in an 8-GPU domain. To find its source, we develop an algorithm that recovers the chain of operations that delays each GPU before a collective. Our algorithm attributes 78% of the tax to variation in the runtimes of matrix-multiplication (GEMM) kernels across GPUs. Since the tax depends on the slowest of many GPUs, our analytical model shows that it grows at least logarithmically with domain size. Thus, a 72-GPU domain requires 3.3x more per-GPU bandwidth than today's simulators predict to keep communication below 12% of step time, and the optimal scale-up fabric should partition a switch's capacity among fewer GPUs with more bandwidth each.

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: We're looking at the paper: "The Cost of Synchronization in GPU Scale-Up Domains".

Title and authors: Tom: So we're looking at this paper called "The Cost of Synchronization in GPU Scale-Up Domains". The title itself tells you right away that we're talking about how communication between GPUs gets slower when you make the domain bigger.

Jane: Right, and the authors are Arjun Devraj Lindsey Bowen Rachee Singh from Cornell University. It’s interesting because they’re looking at something that happens when we try to build really big systems for machine learning.

Lu: It's about how GPUs within a scale-up domain arrive at collective barriers hundreds to thousands of microseconds apart, even when they're all running the same code on the same hardware over a uniform fabric. That’s a huge gap between the design goal and what we see in practice, so it sets up this whole problem.

Meng: I mean, if you're trying to train something massive, that waiting time sounds like wasted compute time just sitting idle. We need to know if this is a hardware limitation or something about how we’re organizing the communication.

Lalam: The authors are showing that when you have an eight-GPU scale-up domain, the synchronization tax can eat up over fifty percent of the total collective communication time, which is a pretty big number for training efficiency <ref:2608.22503#pg1,an 8-GPU scale-up domain>.

Tom: Fifty percent is significant; it means we're not getting the full benefit of our interconnect bandwidth when we try to scale up these clusters. It’s about more than just slow communication; it’s about how much of the time the GPUs are actually doing useful work versus waiting around for a signal.

Jane: So, what they do next is look into what causes this tax and how big it gets as you add more GPUs to that scale-up domain. It moves from describing the problem to figuring out where the problem is coming from.

The paper's summary: Tom: The core idea of this paper, "The Cost of Synchronization in GPU Scale-Up Domains", is that we have two things growing exponentially: interconnect bandwidth and the size of the scale-up domain. These two trends are in tension because they create this synchronization tax.

Jane: So, to put it plainly, as you add more GPUs to a machine learning setup, the time it takes for them all to meet at a collective barrier doesn't stay constant; it grows much faster than we expect.

Lu: They study several hundred thousand collective operations across four different language models and three recent GPU architectures. They found that the identity of the slowest rank changes every time you run a different collective operation, which means it’s not just one bad piece of hardware causing the delay; it’s something systemic in how things are scheduled.

Meng: That's worrying because if it was just a single faulty chip, we could fix that with replacement. If the problem is in the scheduling or the kernel execution variation, then fixing it requires changing how we run things at a fundamental level.

Lalam: They design a graph-based algorithm that looks at per-rank kernel traces to figure out where this overhead comes from, and they found that cross-rank variation in GEMM kernel execution times is responsible for seventy eight percent of this overhead.

Tom: Seventy eight percent of the delay coming from how one GPU handles matrix multiplications differently than another, even when they are on the same hardware and fabric. That's a lot of computational noise hidden inside our training time.

Jane: So, what they show is that this synchronization delay isn't just some random hiccup; it’s tied to the way different parts of the computation interact across those GPUs, especially in tensor parallelism scenarios like tensor parallel parallelism and fully sharded data parallelism.

The paper's improvements: Tom: Now, the authors don't just stop there; they suggest ways to model this variation using extreme value theory. They use that theory to show that this synchronization tax actually grows as the domain size gets bigger. That’s a key finding because it links the problem directly to scaling behavior.

Jane: So, what this means for us is that the synchronization overhead isn't just a fixed cost; it gets worse when we try to run things on much larger GPU clusters. It effectively taxes the benefits of adding more bandwidth.

Lu: They build on that by proposing a bandwidth scaling model. This model shows that as the scale-up domain size grows, these synchronization overheads increase, which means you need less physical interconnect bandwidth than the simple baseline models predicted for huge systems.

Meng: That’s an important implication for deployment. If we design our infrastructure based on older assumptions about how much bandwidth we need, we might be over-provisioning or under-provisioning in a way that doesn't actually match what the actual training time looks like.

Lalam: They find that for a five hundred twelve GPU scale-up domain, you only need two point zero six times less optimal bandwidth compared to an eight GPU domain, which is a concrete number for planning purposes.

Tom: So we’re looking at needing lower interconnect bandwidth requirements as the cluster gets larger, because the synchronization tax eats up that extra potential bandwidth. That’s a really counterintuitive result if you're used to thinking more GPUs mean more bandwidth needed.

Conclusion: Jane: So, wrapping up this paper, "The Cost of Synchronization in GPU Scale-Up Domains" shows us that the synchronization tax is real and it scales with domain size. It’s driven primarily by how much the execution time of matrix multiplication kernels varies between different GPUs.

Tom: The main practical implication for me is that when we design our next generation systems, we can’t just focus on raw interconnect speed anymore; we have to factor in this synchronization tax into our communication budget.

Lu: For future work, they're looking at incorporating their findings into a more complete communication cost model using the Hockney model. This would allow us to calculate the optimal bandwidth requirement by explicitly including that synchronization tax term in the formula for B star.

Meng: From an engineering standpoint, if we can accurately predict this delay, it helps us decide what to deploy and how much hardware to budget for when we move into those massive scale-up domains.

Lalam: I'm confident that by incorporating this synchronization tax into the model, we can get a much more precise prediction of the required bandwidth for any given training setup.

Tom: So, in short, this paper is telling us that as GPU clusters grow larger, the communication bottleneck isn't just about the wires; it’s about waiting around for each other to finish their local work. That's what we learned from "The Cost of Synchronization in GPU Scale-Up Domains".

More episodes

← Home