Backend
AInfer-PD: Communication-Safe In-Place Prefill-Decode Multiplexing for Distributed MoE Rollouts
arXiv cs.DC0 views
arXiv:2609.00993v3 Announce Type: replace Abstract: In agentic reinforcement learning (RL), asynchronous multi-turn trajectories repeatedly introduce continuation prefills (P) while other trajectories remain in decode (D). Efficient rollout completion therefore requires D to advance during long P executions. In-place multiplexing shares devices, model weights, and KV state, but phase-specific communication in distributed MoE introduces two constraints: cross-rank collective ordering and ownership of mutable communication state. Whole-prefill coordination can ensure safe ordering but delays D. DeepEP's normal and low-latency paths suit large P and small D batches, respectively, but safely overlapping them requires separating their mutable communication state. We present AInfer-PD, which extends in-place P/D multiplexing to distributed MoE rollouts. AInfer-PD establishes rank-consistent collective ordering at safe communication boundaries, allowing successive D iterations to advance without waiting for an entire prefill execution to be enqueued or completed. It gives P and D separate communication buffers and protocol state within one DeepEP runtime, allowing P to use the normal path while D retains the low-latency path. Together, these mechanisms preserve useful P/D overlap while sharing model weights and KV storage on the same devices. Across repeated single-node prefill-intensive workloads, AInfer-PD reduces fixed-workload rollout completion time by 7.1-22.5% relative to the same AInfer engine with P/D multiplexing disabled and by 24.8-32.9% relative to SGLang. On two nodes, the reductions are 18.0-35.3% and 18.3-31.8%, respectively. In a same-engine ablation, fine-grained boundaries reduce completion time by a further 8.6-19.8% over whole-prefill asynchronous enqueue.
Source:https://arxiv.org/abs/2609.00993PreviousAngelfish: Leader, DAG, or Anywhere in BetweenNextWhy Kubernetes Cannot Contain a Full Lazy-Pulling Cache: Two Impossibility Results and a Bounded, Honest Design for Model Serving
Related posts
Autoscaling Adds Waiters, Not Ovens. Here's What Actually Breaks at 10x Load
0dev.to
[백엔드 모의면접] 중복 결제는 어떻게 막을까?
0dev.to
Server-Side Pagination Looked Easy. It Took Me Longer Than Any React Concept Yet.
0dev.to
Before You Create an AWS Account, Spend 45 Minutes Learning These Security Tools
0dev.to
Claude “Invalid Signature” in a Thinking Block: How to Debug It Safely
0dev.to
How to catch an expiring or broken SSL certificate before your visitors do
0dev.to