Backend
Twig: Parallel and Memory-Efficient Tree Speculative Decoding for Hybrid-Attention LLMs
arXiv cs.DC0 views
arXiv:2608.01651v2 Announce Type: replace Abstract: Emerging LLMs increasingly interleave full attention with recurrent linear attention to reduce long-context serving costs. Tree speculative decoding could further accelerate these hybrid models, but recurrent linear-attention layers carry parent-dependent states. Existing engines therefore verify proposal trees sequentially and snapshot the full state per node, causing latency and memory to grow with tree and batch sizes. We present Twig, a kernel--runtime co-design for tree speculative decoding of hybrid LLMs. Twig derives a tree-structured factorization of the linear-attention recurrence and realizes it with a resource-efficient GPU kernel that verifies all nodes in parallel, accelerating linear-attention tree verification by 3.4--7.7$\times$. It losslessly represents speculative state updates as token-level factors and reconstructs only the accepted state, reducing transient state memory by 82--99$\times$ and freeing GPU memory for KV caches. We integrate Twig into SGLang, a widely deployed production LLM serving engine, and further optimize its runtime through batch-wide verification budgeting. Across four models, two GPU platforms, and diverse datasets, Twig achieves up to $4.72\times$ the offline decode throughput of autoregressive decoding and up to $2.03\times$ that of the strongest tree-speculative baseline. Under online agent workloads, Twig reduces TTFT and TPOT by up to $67.6\%$ and $49.9\%$, respectively, over the strongest tree-speculative baseline.
Source:https://arxiv.org/abs/2608.01651PreviousWave-Based Dispatch for Circuit Cutting in Hybrid HPC-Quantum SystemsNextLantern: Finding Committable Transactions via Back-Propagation on DAGs
Related posts
Show HN: Retex, a Faster LaTeX Engine
0Hacker News Show
Vulkan 1.4.368 Released With New Extensions From Valve & NVIDIA
0Phoronix
CRA-Durchführungsgesetz & NIS-2 Lieferkettensicherheit (§ 30 BSIG) 2026: BSI-Marktüberwachung, SBOM-Pflicht (TR-03183-2)
0dev.to

Integrating a self-hosted crypto gateway without a plugin: the REST API path
0dev.to
Quoting Failure Numbers Without Lying: A Short Guide for Operators
0dev.to

Why We Stopped Doing One-Shot Scrapes and Built an Agent That Never Forgets
0dev.to