Model quantization
Authorship and approval
[Written with AI] [Approved by a human]
Updated
03.11.2026
Quantization is the process of storing the numbers that make up a model (weights) using fewer bits, e.g., 4 instead of 16. The model then takes up several times less memory and runs faster, at the cost of a minor loss in quality.
18 GB
9B model at 16 bits
6.2 GB
the same model at 4 bits
15 GB
our server RAM
How does it work?
- A model consists of billions of numbers (weights). During training, each typically takes 16 bits, which is 2 bytes.
- A 9B model at 16 bits therefore weighs about 18 GB and will not fit on a regular laptop.
- Quantization rounds weights to a smaller number of possible values: 8 bits cuts the size in half, 4 bits makes it roughly one-third.
- Better methods quantize layers unevenly: more important parts of the model get more bits, less important parts get fewer.
What do Q4, Q8, and K_M mean?
- Q8: 8 bits per weight. Almost no quality loss, but the model is large.
- Q4: approx. 4 bits per weight. The most common choice for a home computer: several times less memory, minor quality loss.
- Q3, Q2: even less memory, but quality drops noticeably: the model makes more mistakes and loses track of the context.
- K_M, K_S: mixed-bit variants of the method (M = medium, S = small). Q4_K_M is a common default choice in Ollama.
- GGUF: the file format for quantized models used by llama.cpp and Ollama.
How much is lost?
At 8 bits the difference is usually unnoticeable, at 4 bits it's small, and at 3 and 2 bits it's significant. The loss depends on the model and the task: a larger model handles quantization better than a small one. Therefore, a 9B model at 4 bits usually performs better than a 4B model at 16 bits, while taking up less memory.
How does our lab do it?
- The lab server has 15 GB RAM and no graphics card. A 9B model at 16 bits (approx. 18 GB) would not fit on it.
- Ornith 1.5 9B in Q4_K_M takes 6.2 GB RAM in our lab and outputs 3.0 tokens per second. Qwen 3.5 4B in Q4_K_M takes 3.1 GB and outputs 5.0 tokens per second.
- Both models in this form processed 2k research papers for our topic classifier. We check the agreement between their labels and human labels on a separate test set.
Related terms
Ollama · Large Language Model (LLM) · Fine-tuning vs RAG · Tokens and Context Window
Questions and answers
Does quantization break the model?
At 8 and 4 bits usually only slightly. Noticeable degradation appears only at 3 and 2 bits, especially in small models.
What to choose: Q4 or Q8?
If the Q8 model fits in memory with room to spare, use Q8. If not, Q4_K_M. It's usually better to pick a larger model in Q4 than a smaller one in Q8.
Can I quantize a model myself?
Yes, llama.cpp has a tool for this. Usually you don't need to: popular models are already available in several quantization versions, and Ollama downloads one of them by default.
Sources
- llama.cpp, repository (GGUF format and quantization)
- Lab: model measurement on i5-2400 and ollama show for B138 classifier models (November 2026)
Next