GGUF quantization shrinks an open language model by storing its weights in fewer bits per number, so a model that needs 16 GB at full precision can run in under 5 GB. It is why a capable model runs on an ordinary laptop, and the letters after a model name, like Q4_K_M, are the recipe used to shrink it.

Key takeaways

  • GGUF is a file format from llama.cpp, and the suffix in a file name, like Q4_K_M, says how the weights inside were compressed.
  • Q4_K_M stores about 4.89 bits per weight, roughly 31 percent of the 16-bit original, according to the llama.cpp quantization README.
  • Down to 4 bits the measured quality loss is small: a January 2026 study found Llama 3.1 8B Instruct scored 69.15 at Q4_K_M against 69.47 at 16-bit.
  • Smaller files also run faster, because generating a token means reading every active weight from memory once: about 72 tokens per second at Q4_K_M against 29 at 16-bit.
  • The practical rule is to run the largest model that fits at Q4_K_M, and to spend spare memory on longer context before stepping up to a higher quant.

Why does quantization matter more now than it did a year ago?

Open models keep getting bigger while our laptops stay the same size. Ant Group open-sourced Ring-2.6-1T, a trillion-parameter agentic model, on 12 November 2026, and Alibaba put Qwen4-Max into preview the same week with a 1 million token context window. Neither will ever run on a laptop in full. Every new open release arrives as a file that has to fit somewhere, and quantization decides whether it fits on your hardware or stays in a data centre.

What do Q4_K_M, Q8_0 and the other suffixes actually mean?

GGUF is a container format created for llama.cpp, also used by Ollama and LM Studio. One file holds the weights, the tokenizer and the model metadata. The suffix in the file name describes how the weights were packed.

The code is simpler than it looks. The number after the Q is roughly the bits per weight, so Q8 is about 8 bits and Q4 is about 4. The K marks the k-quant family, added to llama.cpp in 2023, which groups weights into super-blocks and quantizes the scales themselves. The final letter, S, M or L, says how many of the sensitive tensors get extra bits. Q4_K_M, for example, keeps half of the attention value tensors and half of the feed-forward down-projection tensors at 6-bit, and stores everything else at 4-bit, according to the original k-quant pull request.

The real bits per weight run higher than the names suggest, because the per-block scales take space and the mixed recipes keep some tensors at higher precision. Here is what llama.cpp measured for Llama 3.1 8B:

QuantBits per weightSize against 16-bitF1616.00100%Q8_08.5053%Q6_K6.5641%Q5_K_M5.7036%Q4_K_M4.8931%Q3_K_M4.0025%Q2_K3.1620%

Two other families show up in file names. IQ formats, the i-quants, are built for low bit rates and work best with an importance matrix, a measurement of which weights matter most, taken from a sample of calibration text.

How much quality do you actually lose?

Three kinds of evidence point the same way: almost nothing lost down to 5 or 6 bits, a small loss at 4, a clearer loss at 3, and a steep drop at 2 and below.

The original k-quant work measured perplexity, how surprised a model is by real text, on the original LLaMA 7B. Perplexity rose from 5.9066 at 16-bit to 5.9601 at Q4_K_M, a rise of 0.91 percent. At Q3_K_M it rose 4.1 percent, and at Q2_K it rose 14.7 percent.

A January 2026 study, published as arXiv 2601.14277, ran Llama 3.1 8B Instruct through a benchmark suite at each GGUF level. The average score was 69.47 at 16-bit, 69.41 at Q8_0, 69.36 at Q5_K_M, 69.15 at Q4_K_M, 68.07 at Q3_K_M and 65.49 at Q3_K_S. Down to Q4_K_M the average moved by about a third of a point, which is within noise. Maths took the hardest hit at 3 bits, with GSM8K falling from 77.63 to 68.31 at Q3_K_S.

One caution applies to all of those averages. A paper titled Accuracy is Not All You Need found that compressed models can match the original model accuracy while flipping a meaningful number of individual answers, with right-to-wrong and wrong-to-right changes cancelling out in the average. If your work needs exact output, test the quant on your own tasks and step up a level if you see odd mistakes.

How do you pick the right level for your hardware?

Start with the size maths, because it is simple arithmetic. A file is roughly parameters times bits per weight, divided by 8. An 8 billion parameter model at 16 bits is about 16 GB, at Q8_0 about 8.5 GB, and at Q4_K_M about 4.9 GB.

Context adds memory on top of the weights, so the file size is not the full story. At an 8K context window with a 16-bit KV cache, Qwen3.8 27B needs about 15 GB at Q3_K_M, 18 GB at Q4_K_M, 22 GB at Q5_K_M, 25 GB at Q6_K and 32 GB at Q8_0, according to the gpu.nz VRAM calculator, using data from October 2026.

Quantization also sets speed. When a model generates a token it reads every active weight from memory once, so generation speed is roughly memory bandwidth divided by the size of the weights. The llama.cpp reference numbers for Llama 3.1 8B show the effect: about 29 tokens per second at 16-bit, 51 at Q8_0 and 72 at Q4_K_M.

Our decision rule follows from those numbers. Find the largest model that fits with 10 to 15 percent of memory free, and start it at Q4_K_M. Spend spare memory on context first if you work with long documents or code, then on Q5_K_M or Q6_K. Q8_0 makes sense when memory is plentiful. Below 4 bits, use an importance matrix or a dynamic quant, and only when that gets you a much larger model.

How can you quantize a model yourself?

You do not need to download a pre-quantized file. The llama.cpp tools will quantize any GGUF model you already have:

llama-quantize model-f16.gguf model-q4-k-m.gguf Q4_K_M

For better results at 4 bits and below, build an importance matrix first, by running the full-precision model over a sample of calibration text, then pass it to the quantizer:

llama-imatrix -m model-f16.gguf -f calibration.txt -o model.imatrix llama-quantize --imatrix model.imatrix model-f16.gguf model-q4-k-m.gguf Q4_K_M

Then verify the result instead of trusting the file name. The perplexity tool measures the quantized model against the same text the original was measured on:

llama-perplexity -m model-q4-k-m.gguf -f wikitext.txt

If the perplexity rise is under about 1 percent, the quant is behaving like the published numbers. If it is far above that, step up a level or rebuild with an importance matrix. The KV cache is quantized separately: llama.cpp stores it at 16-bit by default, and the flags -ctk and -ctv switch keys and values to q8_0 or q4_0. In Ollama, the same setting is OLLAMA_KV_CACHE_TYPE.

What about MXFP4, NVFP4 and MLX?

GGUF is not the only 4-bit game in town. MXFP4 stores each weight as a 4-bit floating point number with a shared 8-bit scale per block of 32 values, for about 4.25 bits per weight. OpenAI released gpt-oss-20b and gpt-oss-120b with their mixture-of-experts weights in MXFP4, which is why the 120 billion parameter model fits on a single 80 GB GPU, according to the Hugging Face announcement. If you run gpt-oss, use the MXFP4 file as released instead of requantizing it. NVFP4 is NVIDIA's 4-bit format for Blackwell GPUs, at about 4.5 bits per weight, and NVIDIA reports an accuracy loss of 1 percent or less against FP8 on DeepSeek-R1-0528. On Apple silicon, the MLX framework stores weights at 4 bits by default, in groups of 64, which works out to about 4.5 bits per weight.

What is still unknown?

We should be honest about the gaps. The January 2026 benchmark study covered one 8 billion parameter instruct model, so the exact loss at each level will differ on other models and sizes. Vendor numbers need the same care: Unsloth publishes MMLU scores for its own quants of Gemma 3 27B, 71.5 at BF16 against 71.23 at Q4_K_M, but they remain a vendor's numbers for its own product.

Two questions have no solid published answer yet. The first is how quantization affects agentic work, where a model makes dozens of tool calls in a row and small errors can compound. Unsloth's own documentation says its 1-bit quants should not be used for agentic work, which tells you the vendors are cautious too. The second is how the new 4-bit floating point formats compare with Q4_K_M on the same model and the same tasks.

The practical answer has not changed, though. Quantization is what turns a model release into a model you can actually run, and the numbers say the cost of doing it is small down to 4 bits. Pick the largest model that fits, start at Q4_K_M, and verify it on your own tasks.

Originally published on LogicStack. Run the code examples right in your browser with our free interactive Python and SQL playgrounds — no setup, no sign-up.