Backend
AInfer-PD: Communication-Safe In-Place Prefill-Decode Multiplexing for Distributed MoE Rollouts
arXiv cs.DC0 views
arXiv:2609.00993v3 Announce Type: replace Abstract: In agentic reinforcement learning (RL), asynchronous multi-turn trajectories repeatedly introduce continuation prefills (P) while other trajectories remain in decode (D). Efficient rollout completion therefore requires D to advance during long P executions. In-place multiplexing shares devices, model weights, and KV state, but phase-specific communication in distributed MoE introduces two constraints: cross-rank collective ordering and ownership of mutable communication state. Whole-prefill coordination can ensure safe ordering but delays D. DeepEP's normal and low-latency paths suit large P and small D batches, respectively, but safely overlapping them requires separating their mutable communication state. We present AInfer-PD, which extends in-place P/D multiplexing to distributed MoE rollouts. AInfer-PD establishes rank-consistent collective ordering at safe communication boundaries, allowing successive D iterations to advance without waiting for an entire prefill execution to be enqueued or completed. It gives P and D separate communication buffers and protocol state within one DeepEP runtime, allowing P to use the normal path while D retains the low-latency path. Together, these mechanisms preserve useful P/D overlap while sharing model weights and KV storage on the same devices. Across repeated single-node prefill-intensive workloads, AInfer-PD reduces fixed-workload rollout completion time by 7.1-22.5% relative to the same AInfer engine with P/D multiplexing disabled and by 24.8-32.9% relative to SGLang. On two nodes, the reductions are 18.0-35.3% and 18.3-31.8%, respectively. In a same-engine ablation, fine-grained boundaries reduce completion time by a further 8.6-19.8% over whole-prefill asynchronous enqueue.
Source:https://arxiv.org/abs/2609.00993PreviousAngelfish: Leader, DAG, or Anywhere in BetweenNextWhy Kubernetes Cannot Contain a Full Lazy-Pulling Cache: Two Impossibility Results and a Bounded, Honest Design for Model Serving
Related posts
Cryptographic SARIF v2.1.0 Proofs for Autonomous AI Agents & Desktop Suite Innovations (Day 4)
0dev.to
Cybersecurity Best Practices for Small Businesses
0dev.to
Lean AI Second Brain: Babashka, Git & Local LLMs
0dev.to AI

Inside My AI Marketing Agent: More Than a Prompt
0dev.to AI

Building an Intelligent Assistant: My Architect's View of Amazon Bedrock AgentCore
0dev.to AWS
I indexed 2,606 legally free resources to learn the Indian stock market — here's how the no-build site works
0dev.to WebDev