AI & ML
HYPIC: Position-Independent Caching for Hybrid-Attention LLM Serving
arXiv cs.DC0 views
arXiv:2607.01299v3 Announce Type: replace Abstract: In retrieval-augmented generation (RAG) and agentic LLM serving, prompts are assembled from multiple independent segments into long contexts, making prefill dominate the computation cost. To reduce this cost, position-independent caching (PIC) admits KV reuse for non-contiguous segments across requests, while hybrid-attention models reduce computational complexity using linear attention. However, existing PIC methods break down on hybrid-attention models because their per-token KV-cache reuse primitives do not transfer to per-request recurrent states. In this work, we present HYPIC, the first system to accelerate hybrid-attention LLM serving with position-independent caching. For linear-attention layers, we identify the segment-cumulative transition operator as the missing algebraic primitive and cache it for each segment, enabling near-exact and constant-time state composition. For the remaining full-attention layers, we construct a small seam window to propagate hidden states through the linear-attention layers, so that cross-segment attention can be repaired by recomputation. Finally, HYPIC introduces segment parallelism, which parallelizes cache-miss prefill across instances, turning long cold requests into an accelerable workload. Evaluated across four hybrid-attention models and five workloads, HYPIC reduces time-to-first-token (TTFT) by $3.25\times$ on average and improves QPS by $1.66\times$ over Prefix Cache, while preserving task quality with a $1.71$-point gap from Full Recompute.
Source:https://arxiv.org/abs/2607.01299PreviousEkka: Automated Diagnosis of Silent Errors in LLM InferenceNextBreaking Fault Lines: Unifying TEE-Assisted BFT Consensus in Partially Trusted Worlds
Related posts
Rate-Fidelity Control for Wide-Area Quantum Links
0arXiv cs.NI
Who Needs DRAM? We Have Fiber
0arXiv cs.NI
The Multipath Reliable Connection (MRC) Transport
0arXiv cs.NI
From Network Experience to Subscriber Retention: An Explainable AI Framework for Mobile Operators
0arXiv cs.NI
Spatial Path Superposition: A No-Cloning Perspective on MIMO-Inspired Spatial Diversity
0arXiv cs.NI
Optimization of Joint Beamforming in RIS-aided Multi-carrier and Multi-user MIMO Systems
0arXiv cs.NI