Hierarchical key‑value sharing cuts transformer memory use by about one third while preserving, even improving, downstream performance. The idea is simple: later layers reuse the KV pairs produced by earlier, stride‑aligned layers instead of materialising their own caches. This challenges the prevailing belief that deep models must keep a full per‑layer cache to maintain quality.
Before DepthGraft, cross‑layer connectivity was introduced mainly as an architectural novelty, often adding extra matrix multiplications or memory hops. Adjacent‑layer sharing and global‑anchor schemes have been explored, but they do not provide a systematic regularisation of the KV geometry nor a substantial compression of the cache. Practitioners therefore accepted the memory cost of deep inference as inevitable.
DepthGraft reduces the KV cache by roughly one third. The authors write, “DepthGraft distributes non‑local gradient flow across source layers, while preserving prefill early completion and reducing KV cache by roughly one third” [1]. By reconstructing deep‑stage queries from stride‑aligned shallow‑middle sources, the architecture eliminates independent storage for a large fraction of the transformer depth, yielding a memory footprint that fits comfortably on mid‑range GPUs.
The compression does not come at the expense of accuracy; in fact it delivers measurable gains. On a 16 B Mixture‑of‑Experts model, DepthGraft achieves a 7.0 % absolute improvement on MMLU‑Pro and lifts CMMLU and C3 by 6.6 % and 6.4 % respectively [1]. Similar upticks appear across dense models from 0.5 B to several billion parameters, indicating that the hierarchical reuse sharpens key retrieval directions while enriching value subspaces.
The results leave open how the approach scales beyond 16 B parameters or to contexts far longer than the 128 K tokens evaluated in the paper’s README. The design relies on stride‑aligned sharing topology; mismatched layer depths or irregular transformer stacks may require bespoke engineering. Moreover, although per‑token overhead stays below one percent, latency-sensitive deployments should profile the added reconstruction step.
If hierarchical KV sharing lives up to these early findings, inference pipelines could increase effective model size or context length within existing memory budgets without upgrading hardware. Benchmarks that previously capped at a given memory budget—such as MMLU‑Pro runs on a single A100—should be rerun with DepthGraft enabled, and production services ought to consider swapping vanilla caches for the shared variant as the new default.