AI & ML
The Cost of Synchronization in GPU Scale-Up Domains
arXiv cs.DC0 views
arXiv:2608.22503v2 Announce Type: replace Abstract: GPU scale-up domains are the building blocks of machine learning infrastructure. Architects provision the size and bandwidth in these domains assuming all GPUs reach a collective operation at the same time. However, we find that this assumption does not hold in practice. Across 300,000 collectives from training four language models on A100, H100, and H200 domains, GPUs reach the same collective hundreds to thousands of microseconds apart, despite running identical kernels. Since a collective cannot complete until every GPU contributes its data, GPUs that arrive early wait for the slowest one. We call this waiting time the synchronization tax, and it consumes nearly 40% of tensor-parallel communication time in an 8-GPU domain. To find its source, we develop an algorithm that recovers the chain of operations that delays each GPU before a collective. Our algorithm attributes 78% of the tax to variation in the runtimes of matrix-multiplication (GEMM) kernels across GPUs. Since the tax depends on the slowest of many GPUs, our analytical model shows that it grows at least logarithmically with domain size. Thus, a 72-GPU domain requires 3.3x more per-GPU bandwidth than today's simulators predict to keep communication below 12% of step time, and the optimal scale-up fabric should partition a switch's capacity among fewer GPUs with more bandwidth each.
Source:https://arxiv.org/abs/2608.22503PreviousProgrammable Probabilistic Computer with 1,000,000 p-bitsNextA Flat KV Cache Storage Architecture for LLM Serving
Related posts
Designing a useful preview for an AI-generated comic
0dev.to
AI Agents: What They Are and Why They Matter
0dev.to
GPT-6.2 Astra Was Easier to Jailbreak, UK Testing Found
0dev.to
iFixit Says Apple’s HomeView Breaks Easily. Here’s Why I’m Not Worried
0CNET News
Scout Motors Is Using AI Earpieces to Train Factory Workers
0CNET News
'Grand Theft Auto VI' is here. What to know about the characters, music and all the hype
0Phys.org Tech