Backend
DexAttention: Distributed Sparse Attention through Rapid Online Planning and Persistent GPU Execution
arXiv cs.DC0 views
arXiv:2611.23512v1 Announce Type: new Abstract: Sparse attention reduces computation, but distributed execution may still transfer unnecessary data and incur substantial per-operation overhead. Mask-aware assignment and scheduling can reduce these costs, but replanning for dynamic masks must be cheaper than the execution time it saves. We present DexAttention, a distributed sparse attention system that co-designs fast online planning with persistent execution. Its assigner uses GPU-accelerated, topology-aware clustering to balance computational load and reduce communication. A lightweight greedy scheduler orders coarse communication and computation operations, while a persistent kernel on each GPU enforces fine-grained dependencies and dispatches ready tasks. This division keeps detailed synchronization out of planning while enabling overlap without repeated attention launches. The framework supports static and dynamic per-head masks across multiple attention compute kernels. On a four-node, 16-GPU cluster, DexAttention achieves a $1.39\times$ geometric-mean end-to-end training speedup over the fastest applicable baseline across 11 configurations. For dynamic video attention, it achieves $2.11$--$2.37\times$ speedups over Ring Attention including planning.
Source:https://arxiv.org/abs/2611.23512PreviousEnd-to-End Multi-Task Learning for Adjustable Joint Noise Reduction and Hearing Loss CompensationNextStephen: Verifying Sequencer-Aided Coordination Systems with Cross-View Prophecy
Related posts
Download Website as HTML: Get a Runnable Project From a Figma Capture
0dev.to Design
A bilingual article’s canonical URL changes from a draft decision to an API result
0dev.to Discuss
Signs of Wear: Evaluating Intelligent Speed Assistance Under Synthetic Adverse Conditions
0arXiv eess.IV
SWIMSS: An Open Multi-Center SwinUNETR Model for Multiple Sclerosis White Matter Lesion Segmentation
0arXiv eess.IV
SurgCBVR: A Multi-Domain Benchmark for Content-Based Surgical Video-to-Video Retrieval
0arXiv eess.IV
Where Was the Author of The Ugly Duckling Born? What 11 LLMs Taught Me About Connecting Facts
0dev.to Machine Learning