AI & ML
XM-Sparse: High-Throughput Sparse-Attention LLM Serving over Scale-up Memory Pools
arXiv cs.DC0 views
arXiv:2611.22280v1 Announce Type: new Abstract: Sparse attention has been increasingly adopted by production LLMs for long-context inference, yet serving systems must still retain the full KV history, making GPU HBM capacity a key limit on serving concurrency. Existing systems offload the KV cache to host memory, but suffer from the limited bandwidth of PCIe and the limited capacity of host memory. Emerging scale-up memory pools (e.g., X-Mem) relieve both limits and open new opportunities for sparse-attention offloading. In this paper, we first conduct an in-depth analysis and reveal that, even with scale-up memory pools, decode throughput remains bounded by HBM capacity, as hot KV buffers and the fully resident indexer key (K-index) cache occupy substantial HBM. Moreover, the two caches exhibit distinct access patterns. Based on these insights, we present XM-Sparse, the first sparse-attention offloading system over scale-up memory expansion. The core idea of XM-Sparse is to offload both caches aggressively while matching each with an I/O mechanism tailored to its access pattern. Specifically, for dynamically selected KV entries, XM-Sparse fuses TMA-based retrieval with attention in a tile-level pipeline to hide on-demand retrieval latency. For predictable K-index accesses, it prefetches pages via SM-efficient GPU-initiated DMA and coordinates prefetching with latency-critical operations to avoid interfering with decoding. We have implemented XM-Sparse atop SGLang, and our experiments on DeepSeek-V3.2 and GLM-5.3 with an emulated scale-up memory pool show that XM-Sparse delivers up to 4.98$\times$ and 4.52$\times$ higher decode throughput than PCIe-based ECHO and HiSparse, respectively, and up to 1.34$\times$ over a naive integration on the same platform, across 16K-1M contexts.
Source:https://arxiv.org/abs/2611.22280PreviousTAILROLL: Tail-Aware Rollout Acceleration for RL-Based LLM Post-TrainingNextCheckCo: Checkpoint-Coordinated Load Balancingfor Skewed Event Streams at Scale
Related posts
Web-Perf Wednesday 018 – Soft Navigations Lift the CrUX Pass Rate
0CSS Wizardry

‘It’s a risk’: How Rockstar built the most detailed (and romantic) GTA yet
0The Verge
China dominates battery graphite, but this startup could break the monopoly
0TechCrunch
AI visibility: mentions, recommendations and citations need separate checks
0dev.to
Background AI Makes Inference a Scheduling Problem
0dev.to

DeepSeek vs Claude vs Kimi vs Codex vs Gemini: A 70-Call AI Ethics Test
0dev.to