aipoc-lab
[AI GLOSSARY][MODELS][LLM][OPEN SOURCE]

Model quantization

Authorship and approval

[Written with AI] [Approved by a human]

Updated

03.11.2026

Quantization is the process of storing the numbers that make up a model (weights) using fewer bits, e.g., 4 instead of 16. The model then takes up several times less memory and runs faster, at the cost of a minor loss in quality.

18 GB

9B model at 16 bits

6.2 GB

the same model at 4 bits

15 GB

our server RAM

How does it work?

  1. A model consists of billions of numbers (weights). During training, each typically takes 16 bits, which is 2 bytes.
  2. A 9B model at 16 bits therefore weighs about 18 GB and will not fit on a regular laptop.
  3. Quantization rounds weights to a smaller number of possible values: 8 bits cuts the size in half, 4 bits makes it roughly one-third.
  4. Better methods quantize layers unevenly: more important parts of the model get more bits, less important parts get fewer.

What do Q4, Q8, and K_M mean?

  • Q8: 8 bits per weight. Almost no quality loss, but the model is large.
  • Q4: approx. 4 bits per weight. The most common choice for a home computer: several times less memory, minor quality loss.
  • Q3, Q2: even less memory, but quality drops noticeably: the model makes more mistakes and loses track of the context.
  • K_M, K_S: mixed-bit variants of the method (M = medium, S = small). Q4_K_M is a common default choice in Ollama.
  • GGUF: the file format for quantized models used by llama.cpp and Ollama.

How much is lost?

At 8 bits the difference is usually unnoticeable, at 4 bits it's small, and at 3 and 2 bits it's significant. The loss depends on the model and the task: a larger model handles quantization better than a small one. Therefore, a 9B model at 4 bits usually performs better than a 4B model at 16 bits, while taking up less memory.

How does our lab do it?

  • The lab server has 15 GB RAM and no graphics card. A 9B model at 16 bits (approx. 18 GB) would not fit on it.
  • Ornith 1.5 9B in Q4_K_M takes 6.2 GB RAM in our lab and outputs 3.0 tokens per second. Qwen 3.5 4B in Q4_K_M takes 3.1 GB and outputs 5.0 tokens per second.
  • Both models in this form processed 2k research papers for our topic classifier. We check the agreement between their labels and human labels on a separate test set.

Related terms

Ollama · Large Language Model (LLM) · Fine-tuning vs RAG · Tokens and Context Window

Questions and answers

Does quantization break the model?

At 8 and 4 bits usually only slightly. Noticeable degradation appears only at 3 and 2 bits, especially in small models.

What to choose: Q4 or Q8?

If the Q8 model fits in memory with room to spare, use Q8. If not, Q4_K_M. It's usually better to pick a larger model in Q4 than a smaller one in Q8.

Can I quantize a model myself?

Yes, llama.cpp has a tool for this. Usually you don't need to: popular models are already available in several quantization versions, and Ollama downloads one of them by default.

Sources

Next