KV Cache Math: Llama 3.1 8B Needs 16 GiB Just to Hold 128K Tokens

The model fit. That was the part that confused me.

Llama 3.1 8B in bf16 is about 16 GB of weights. My card had 24 GB. I set max_model_len=131072 because the model card says 128K context, hit run, and vLLM refused to start. Something about the max sequence length being larger than the number of tokens that can be stored in the KV cache.

I had 8 GB of headroom. How was that not enough for one request?

Because the KV cache math for Llama 3.1 8B says a single 128K-token sequence needs 16 GiB on its own. Not the weights. Just the cache. This post is the arithmetic I should have done before I picked that number, and it applies to every transformer you will ever self-host.

TL;DR

• KV cache size per token = 2 × layers × KV heads × head_dim × bytes per value. For Llama 3.1 8B in bf16 that is 2 × 32 × 8 × 128 × 2 = 131,072 bytes, exactly 128 KiB per token.
• 128K tokens × 128 KiB = 16 GiB for one full-length sequence, on top of ~16 GB of weights. A 24 GB card cannot hold both.
• vLLM checks this at startup: if one sequence of max_model_len cannot fit in the KV memory left after weights, it refuses to run.
• Fixes, cheapest first: lower max_model_len to what you actually use, set kv_cache_dtype="fp8" (halves the cache), quantize the weights to free memory for cache, or move to a bigger GPU.
• Concurrency is the same math: total KV cache tokens ÷ tokens per request ≈ how many requests run at once.

What is actually stored in the KV cache?

The KV cache stores the key and value vectors for every past token, in every layer, so the model does not recompute them each time it generates a new token. Without it, generating token 10,001 would mean re-running attention projections for the previous 10,000 tokens. With it, you compute K and V once per token per layer and keep them around.

That "keep them around" is the bill. Every layer has its own K and V. Every KV head has its own vectors. Every token adds one more row to every one of those.

Picture it as 32 shelves (layers), each with two rows (K and V). Each new token drops a box on every shelf, in both rows. Nothing ever gets removed until the request finishes.

How do you calculate KV cache size per token?

Multiply four numbers from the model's config.json and the dtype size: 2 × num_hidden_layers × num_key_value_heads × head_dim × bytes. The 2 is for K and V. Here is the version I now keep in a scratch file:

import json def kv_bytes_per_token(config_path, bytes_per_value=2): cfg = json.load(open(config_path)) layers = cfg["num_hidden_layers"] kv_heads = cfg.get("num_key_value_heads", cfg["num_attention_heads"]) head_dim = cfg.get("head_dim") or cfg["hidden_size"] // cfg["num_attention_heads"] return 2 * layers * kv_heads * head_dim * bytes_per_value per_tok = kv_bytes_per_token("Llama-3.1-8B/config.json") print(per_tok) # 131072 print(per_tok * 131072 / 2**30) # 16.0 GiB for one 128K sequence 

Llama 3.1 8B: 32 layers, 8 KV heads, head_dim 128 (4096 hidden / 32 attention heads), bf16 = 2 bytes.

2 × 32 × 8 × 128 × 2 = 131,072 bytes = 128 KiB per token 

Scale that up and it stops looking small:

It is linear. Double the context, double the cache. There is no clever compression hiding in there by default.

Why does vLLM refuse to start with max_model_len=131072?

vLLM refuses because, after loading weights and reserving working memory, the remaining KV cache budget is smaller than one sequence of max_model_len tokens. It does this check up front so you fail at boot instead of at 3 a.m. when the first long document arrives.

Here is the rough budget on a 24 GB card, with the default gpu_memory_utilization=0.9:

usable by vLLM ≈ 0.9 × 24 GB ≈ 21.6 GB weights (bf16) ≈ 8.03B × 2 B ≈ 16.1 GB activations, graphs, overhead ≈ 1-2 GB left for KV cache ≈ 3.5-4.5 GB 

At 128 KiB per token, 4 GiB holds about 32,768 tokens. Total. Across all requests. My 131,072 ask was four times what existed.

The important mental shift: the KV cache budget is a pool of tokens, not a pool of requests. vLLM's PagedAttention hands out that pool in small blocks (16 tokens each by default), so a short request only takes what it uses. But the pool has a hard size, and max_model_len has to fit inside it at least once.

Why is Llama 3.1 8B's cache "only" 128 KiB per token?

Because of grouped-query attention (GQA). Llama 3.1 8B has 32 query heads but only 8 KV heads, so groups of 4 query heads share one set of K and V. The cache stores 8 heads, not 32.

Compare it with Llama 2 7B, which used classic multi-head attention with 32 KV heads:

Llama 2 7B (MHA): 2 × 32 × 32 × 128 × 2 = 512 KiB per token Llama 3.1 8B (GQA): 2 × 32 × 8 × 128 × 2 = 128 KiB per token 

Same layer count, same head size, 4x less cache. Without GQA, a 128K-token sequence on an 8B-class model would need 64 GiB of cache. GQA is a big part of why long context is practical at all.

Bigger models do not get a pass. Llama 3.1 70B has 80 layers and still 8 KV heads:

2 × 80 × 8 × 128 × 2 = 327,680 bytes = 320 KiB per token 128K tokens → 40 GiB of KV cache for one sequence 

How do you reduce KV cache memory in vLLM?

You have four levers: shorter max_model_len, an fp8 KV cache, quantized weights, and more memory headroom. In order of how often I reach for them:

1. Set max_model_len to what your traffic needs. I looked at my real prompts. Almost nothing went past 20K tokens. Advertising 128K was vanity.

2. Store the cache in fp8. One byte per value instead of two. Per token drops from 128 KiB to 64 KiB, so the same pool holds twice the tokens. Quality impact depends on the model and the task, so run your own evals before trusting it on anything that matters.

3. Quantize the weights. A 4-bit AWQ build of an 8B model is roughly a third the size of bf16. Every GB you take out of the weights goes straight into the KV pool.

4. Raise gpu_memory_utilization carefully. Going from 0.90 to 0.95 on 24 GB is about 1.2 GB more, which is around 9,600 more bf16 tokens. Do not push it so high that something else on the GPU (a desktop session, a second process) gets starved.

The config I ended up with:

from vllm import LLM llm = LLM( model="meta-llama/Llama-3.1-8B-Instruct", max_model_len=32768, kv_cache_dtype="fp8", gpu_memory_utilization=0.92, ) 

Same flags on the CLI:

vllm serve meta-llama/Llama-3.1-8B-Instruct \ --max-model-len 32768 \ --kv-cache-dtype fp8 \ --gpu-memory-utilization 0.92 

Then read the startup log. vLLM prints the KV cache capacity in tokens and a "maximum concurrency" estimate for your max_model_len. That one line tells you more than any blog post, including this one.

How does KV cache size limit concurrency and latency?

Concurrency is the KV token pool divided by tokens per request. If the pool holds 64K tokens and your typical request is prompt plus output of 8K, you get about 8 requests running at once. Request number 9 waits in the queue, or vLLM preempts a running one and recomputes it later. Both show up as latency spikes that look like "the server is slow" in your dashboards.

There is a second cost that surprised me more. Decoding is memory-bandwidth bound: for each new token, the GPU reads the weights and the entire KV cache for that sequence. At 128K tokens in bf16, that is 16 GiB of cache read per generated token, about the same as the 16 GB of weights. So at full context, the cache roughly doubles the bytes moved per step. Long context is not just a memory problem, it is a tokens-per-second problem.

Practical rules I now follow:

• Budget concurrency with real request lengths, not the model card's max.
• Watch the gap between prompt length and max_model_len. A cap way above real usage buys you nothing except a failed boot.
• If you need a few very long requests and many short ones, consider routing long ones to a separate instance with its own settings.

The cheat sheet

bytes/token = 2 × layers × kv_heads × head_dim × dtype_bytes cache = bytes/token × tokens pool ≈ (VRAM × gpu_memory_utilization) − weights − overhead concurrency ≈ pool / (bytes/token × avg_tokens_per_request) 

Four lines. I wish I had taped them to my monitor before I spent an evening blaming vLLM for a problem that was pure arithmetic.

So why does Llama 3.1 8B need 16 GiB for 128K tokens?

The KV cache math for Llama 3.1 8B is 2 (K and V) × 32 layers × 8 KV heads × 128 head dimensions × 2 bytes in bf16, which is 128 KiB per token. Multiply by 131,072 tokens and one full-context sequence needs exactly 16 GiB of KV cache, separate from the roughly 16 GB of weights. On a 24 GB GPU that does not fit, so vLLM refuses to start with max_model_len=131072. Lower max_model_len to your real usage, switch to kv_cache_dtype="fp8" to halve the cache, or quantize the weights to give the cache more room.

Written by the developer behind Preterview, an interview prep platform.