AI & ML
The Cost of Synchronization in GPU Scale-Up Domains
arXiv cs.DC0 views
arXiv:2608.22503v2 Announce Type: replace Abstract: GPU scale-up domains are the building blocks of machine learning infrastructure. Architects provision the size and bandwidth in these domains assuming all GPUs reach a collective operation at the same time. However, we find that this assumption does not hold in practice. Across 300,000 collectives from training four language models on A100, H100, and H200 domains, GPUs reach the same collective hundreds to thousands of microseconds apart, despite running identical kernels. Since a collective cannot complete until every GPU contributes its data, GPUs that arrive early wait for the slowest one. We call this waiting time the synchronization tax, and it consumes nearly 40% of tensor-parallel communication time in an 8-GPU domain. To find its source, we develop an algorithm that recovers the chain of operations that delays each GPU before a collective. Our algorithm attributes 78% of the tax to variation in the runtimes of matrix-multiplication (GEMM) kernels across GPUs. Since the tax depends on the slowest of many GPUs, our analytical model shows that it grows at least logarithmically with domain size. Thus, a 72-GPU domain requires 3.3x more per-GPU bandwidth than today's simulators predict to keep communication below 12% of step time, and the optimal scale-up fabric should partition a switch's capacity among fewer GPUs with more bandwidth each.
Source:https://arxiv.org/abs/2608.22503PreviousProgrammable Probabilistic Computer with 1,000,000 p-bitsNextA Flat KV Cache Storage Architecture for LLM Serving
Related posts
AI Agent Observability: What to Log Before Production
0dev.to
AI Agent Evaluation: How to Test Workflows That Change Their Answers
0dev.to
paper-orchestra: A Multi-Agent Pipeline That Turns Experiment Logs Into LaTeX Submissions
0dev.to
Kerberos Through the HTTPS Tunnel: Pre-Auth Capture via the KDC Proxy
0dev.to
We checked 100 well-known sites for AI-crawler readiness: llms.txt, robots.txt, schema, sitemaps
0dev.to
We asked ChatGPT, Perplexity and Gemini which software to buy. Here's who they named first
0dev.to