AI & ML
HYPIC: Position-Independent Caching for Hybrid-Attention LLM Serving
arXiv cs.DC0 views
arXiv:2607.01299v3 Announce Type: replace Abstract: In retrieval-augmented generation (RAG) and agentic LLM serving, prompts are assembled from multiple independent segments into long contexts, making prefill dominate the computation cost. To reduce this cost, position-independent caching (PIC) admits KV reuse for non-contiguous segments across requests, while hybrid-attention models reduce computational complexity using linear attention. However, existing PIC methods break down on hybrid-attention models because their per-token KV-cache reuse primitives do not transfer to per-request recurrent states. In this work, we present HYPIC, the first system to accelerate hybrid-attention LLM serving with position-independent caching. For linear-attention layers, we identify the segment-cumulative transition operator as the missing algebraic primitive and cache it for each segment, enabling near-exact and constant-time state composition. For the remaining full-attention layers, we construct a small seam window to propagate hidden states through the linear-attention layers, so that cross-segment attention can be repaired by recomputation. Finally, HYPIC introduces segment parallelism, which parallelizes cache-miss prefill across instances, turning long cold requests into an accelerable workload. Evaluated across four hybrid-attention models and five workloads, HYPIC reduces time-to-first-token (TTFT) by $3.25\times$ on average and improves QPS by $1.66\times$ over Prefix Cache, while preserving task quality with a $1.71$-point gap from Full Recompute.
Source:https://arxiv.org/abs/2607.01299PreviousEkka: Automated Diagnosis of Silent Errors in LLM InferenceNextBreaking Fault Lines: Unifying TEE-Assisted BFT Consensus in Partially Trusted Worlds
Related posts
Using profiles of cognitive capability to assess AI suitability for workplace tasks
0arXiv cs.HC
Metamaterial tactile displays with programmable virtual pixels
0arXiv cs.HC
MAIA: Multi-Agent Intent Articulation for Requirement Discovery in Art Commissions
0arXiv cs.HC
Patients With Personality: Realistic Patient Simulation through Controlled Diversity and Selective Disclosure
0arXiv cs.HC
Rethinking the UI of GenUI: A Tale of Two Designs
0arXiv cs.HC
AromaGen: Interactive Generation of Rich Olfactory Experiences with Multimodal Language Models
0arXiv cs.HC