TL;DR
A 4B model quantized to Q4_K_M is a 2.50 GB file, yet the same model with a default f16 KV cache at 32K context needs 4.50 GiB just for the cache. The cache, not the weights, sets your RAM ceiling on a phone or laptop. Qwen3-4B has 36 layers, 8 KV heads and head dimension 128, so every token costs 147,456 bytes in f16, 78,336 in q8_0 and 41,472 in q4_0. Below is the byte math, the config fields to read, and a rule for choosing the cache type before you ship.
Why does the KV cache matter more than the quantized weights on-device?
Weights are paid once. The KV cache grows with every token in the context window and is paid per session. For short chats it is small, but long-context features (document Q&A, agents that keep tool output in context) push it past the weight file quickly.
Numbers here come from the public config.json of Qwen/Qwen3-4B and the file sizes listed on the Qwen/Qwen3-4B-GGUF repository, read from the Hugging Face API on 2026-11-20.
What are the fields that set KV cache size?
From the config: num_hidden_layers = 36, num_key_value_heads = 8, head_dim = 128. The model has 32 attention heads, but grouped-query attention stores only 8 K/V heads, a 4x saving over full multi-head storage.
layers, kv_heads, head_dim = 36, 8, 128 elems_per_token = 2 * layers * kv_heads * head_dim # K and V print(elems_per_token) # 73,728 bytes_per_elem = {"f16": 2.0, "q8_0": 34/32, "q4_0": 18/32} for name, b in bytes_per_elem.items(): per_tok = elems_per_token * b print(name, int(per_tok), round(per_tok * 32768 / 2**30, 2), "GiB at 32K") The q8_0 and q4_0 figures use llama.cpp block layouts: 32 values share one scale, so q8_0 is 34 bytes per 32 values and q4_0 is 18 bytes per 32 values.
How big is the KV cache at each context length?
| Context | f16 (GiB) | q8_0 (GiB) | q4_0 (GiB) |
|---|---|---|---|
| 4,096 | 0.56 | 0.30 | 0.16 |
| 8,192 | 1.12 | 0.60 | 0.32 |
| 16,384 | 2.25 | 1.20 | 0.63 |
| 32,768 | 4.50 | 2.39 | 1.27 |
| 40,960 (model max in config) | 5.62 | 2.99 | 1.58 |
The growth is linear: 144 KiB per token in f16. Doubling the context doubles the cache, while the weight file stays fixed.
How big are the weight files?
The GGUF repository lists these sizes in bytes (shown in GB):
| File | Size (GB) |
|---|---|
| Q4_K_M | 2.50 |
| Q5_K_M | 2.89 |
| Q6_K | 3.31 |
| Q8_0 | 4.28 |
Going from Q4_K_M to Q8_0 adds 1.78 GB of weights. Moving the KV cache from f16 to q8_0 at 32K saves 2.11 GiB, which is more than the gap between Q4_K_M and Q6_K (0.81 GB). On a long-context workload, the cache type is the larger lever.
What is the total budget for a 32K session?
Weights plus KV cache, before activations and runtime overhead:
- Q4_K_M + f16 cache: 2.33 GiB + 4.50 GiB = 6.83 GiB
- Q4_K_M + q8_0 cache: 2.33 GiB + 2.39 GiB = 4.72 GiB
- Q4_K_M + q4_0 cache: 2.33 GiB + 1.27 GiB = 3.60 GiB
An 8 GB device that must also hold the OS and the app is comfortable only in the last two rows. Run llama-server with --cache-type-k q8_0 --cache-type-v q8_0 (a quantized V cache needs flash attention enabled in llama.cpp) and measure resident memory on the real device.
How should you choose the cache type before shipping?
- Compute bytes per token from the config, as above.
- Multiply by the longest context your product will actually allow, not the model maximum.
- Start with q8_0 for K and V. Compare your own task outputs against f16 on a fixed prompt set.
- Go to q4_0 only if the RAM budget forces it, and re-run the same prompt set.
- Cap the context in the app so the budget cannot be exceeded by user input.
This is the same discipline we apply when we publish on-device builds at Ginigen: write the memory budget down per context length, then test the target device.
FAQ
Does a lower weight quantization shrink the KV cache?
No. Weight quantization changes the file size. The KV cache depends on layers, KV heads, head dimension, context length and the cache type.
Why is Qwen3-4B's cache smaller than a 32-head model's would be?
Grouped-query attention stores 8 K/V heads instead of 32, which is a 4x reduction in cache per token.
Is a q8_0 cache lossless?
No, it is an approximation. Measure it on your own prompts against f16 before you ship.
Where do I find the numbers for another model?
Read num_hidden_layers, num_key_value_heads and head_dim (or hidden size divided by attention heads) from its config.json, then apply the formula above. Models with hybrid or sliding-window layers need a per-layer-type count.
Do these numbers include activations?
No. Add a margin for compute buffers and the runtime, and verify with a real memory measurement.
Further reading
- Why a Sparse 35B MoE Runs Faster on a CPU Than a Dense 27B Model
- LFM3-1.7B On-Device: The 256K KV Cache Math, Which GGUF to Ship, and the Safety Test to Run First
