AI & ML
HYPIC: Position-Independent Caching for Hybrid-Attention LLM Serving
arXiv cs.DC0 views
arXiv:2607.01299v3 Announce Type: replace Abstract: In retrieval-augmented generation (RAG) and agentic LLM serving, prompts are assembled from multiple independent segments into long contexts, making prefill dominate the computation cost. To reduce this cost, position-independent caching (PIC) admits KV reuse for non-contiguous segments across requests, while hybrid-attention models reduce computational complexity using linear attention. However, existing PIC methods break down on hybrid-attention models because their per-token KV-cache reuse primitives do not transfer to per-request recurrent states. In this work, we present HYPIC, the first system to accelerate hybrid-attention LLM serving with position-independent caching. For linear-attention layers, we identify the segment-cumulative transition operator as the missing algebraic primitive and cache it for each segment, enabling near-exact and constant-time state composition. For the remaining full-attention layers, we construct a small seam window to propagate hidden states through the linear-attention layers, so that cross-segment attention can be repaired by recomputation. Finally, HYPIC introduces segment parallelism, which parallelizes cache-miss prefill across instances, turning long cold requests into an accelerable workload. Evaluated across four hybrid-attention models and five workloads, HYPIC reduces time-to-first-token (TTFT) by $3.25\times$ on average and improves QPS by $1.66\times$ over Prefix Cache, while preserving task quality with a $1.71$-point gap from Full Recompute.
Source:https://arxiv.org/abs/2607.01299PreviousEkka: Automated Diagnosis of Silent Errors in LLM InferenceNextBreaking Fault Lines: Unifying TEE-Assisted BFT Consensus in Partially Trusted Worlds
Related posts

Call of Duty: Modern Warfare 4 composer says multiplayer music is usually 'an afterthought' and 'cobbled together' from existing campaign music — 'This is the first time we've specifically composed a
0TechRadar

GPT-Image-2.6 Outputs SVG and GIFs. Unlike 2.5, You Can't Run It Through Batch
0dev.to

Designing a Form That Asks for Birth Date, Time and Place (No Code, Just the Tricky Parts)
0dev.to UX
메타 Quest 2 OS 업데이트 종료, 약시 훈련 루틴에는 무슨 일이 생기나
0dev.to News
How to Put a Logo in a QR Code (and Make Sure It Still Scans)
0dev.to Design
Why Luxury Fine Jewelry Brands Are Ditching Physical Video Shoots for 3D CAD Commercials in 2026
0dev.to Design