Backend
ReKV: Overlapping Quantized KV Transfer and Suffix Recomputation
arXiv cs.DC0 views
arXiv:2611.08770v1 Announce Type: new Abstract: Prefill--decode disaggregation moves a prompt's key--value (KV) cache between workers before the first token can be generated. Reducing this handoff to either cache compression or recomputation misses a useful choice: which layers to transmit, at what precision, and which layers to execute again on the decode worker. ReKV transfers a quantized prefix KV cache and one BF16 boundary state, then reconstructs a contiguous suffix while the prefix is in flight. In a model-shaped synthetic benchmark on cross-node H100s, the Q8 configuration meeting separately evaluated quality criteria reduces 8K handoff latency by $2.36\times$, $2.84\times$, and $3.65\times$ over BF16 transfer for Qwen2.5-7B, Qwen2.5-3B, and Qwen3-8B. It is also faster than the best measured transfer-only or recompute-only action at those points. A separate real-weight BF16 recomputation experiment reduces prefill-to-token-return time by $1.16$--$1.63\times$. The preferred action changes with prompt length and link bandwidth.
Source:https://arxiv.org/abs/2611.08770PreviousThe GPU Needs an Operating SystemNextAdaptive Scheduling Pipeline for Multi-Instance Asynchronous Reinforcement Learning
Related posts
From One Sentence to a Working App: What Vibe Coding Gets Right — and Where It Still Needs You
0dev.to
A button people press 57 times per visit: building a no-backend tap counter that doesn't lose counts
0dev.to
The 15-Day Clock: Could You Reconstruct an AI Incident Before the Deadline?
0dev.to
DOBI: machine identity, DAMs, and x402 payments as a device workflow
0dev.to

When "Not Found" Doesn't Mean "Never Happened"
0dev.to
Blameless Postmortems: A Leader’s Guide to Learning From Incidents
0dev.to