In plain words
An AI model is, at heart, billions of numbers learned in training. Each is usually stored with lots of decimal places, which takes space. A model with 8 billion numbers stored at the usual 16-bit precision needs about 16 GB of memory, more than most laptops can spare.
Quantization rounds those numbers. Instead of 0.7342, the model stores the nearest of a small set of allowed values, such as 0.733. With 4 bits, each number can be one of only 16 values. The same 8 billion numbers now fit in about 4 GB.
Surprisingly, models barely notice moderate rounding. At 8 bits, quality is usually almost unchanged; at 4 bits it drops a little. Push much lower and answers start to get worse.
Quantization is why people can run capable open models on their own laptops and phones.
Everyday example
Rounding prices at a sabzi market
A vegetable seller's scale shows 0.483 kg of tomatoes, and the exact price works out to ₹19.32. Nobody bothers with the paise; it becomes ₹19 or ₹20. The bill is nearly right and far quicker to settle. Quantization does this to every number inside a model: rounds it to a simpler value that is almost as good.
Where the example breaks: One rounded bill hardly matters. A model makes billions of rounded calculations in a row, and small errors can add up. That is why careful methods choose which numbers to round, and how, instead of rounding everything bluntly.
How it actually works
Quantization changes how a model's parameters (and sometimes its working numbers) are stored, not what the model is.
Fewer bits per number
Models are usually trained in 16- or 32-bit floating point. Quantization maps each value onto a small grid of levels: 256 levels for 8-bit, 16 for 4-bit. A scale factor, stored alongside a block of numbers, converts them back when needed.
Memory is the main win
Memory for weights is roughly parameters × bits ÷ 8. For 8 billion parameters: 16 GB at 16-bit, 8 GB at 8-bit, 4 GB at 4-bit, plus small extras for the scale factors. Less data to move also makes inference faster on most hardware.
Handling the awkward numbers
A few values in large models are unusually big, and rounding them badly hurts quality. Methods like LLM.int8() (2022) keep those outliers at higher precision; GPTQ (2022) adjusts the remaining weights to cancel out rounding error.
After training or during it
Most quantization is done after training, which is quick. Quantization-aware training teaches the model to expect low precision from the start, and holds up better at very low bit counts. Apple used it to compress its roughly 3 billion parameter on-device model to 2 bits per weight in 2025.
File formats you may see
GGUF, created by the developer of the llama.cpp project, is a popular format for quantized models. Names like Q4_K or Q8_0 indicate roughly 4 or 8 bits per weight.
- 8-bit at scale
- LLM.int8() (Dettmers et al., August 2022) halved inference memory for models up to 175B parameters with no loss in their tests
- 3 to 4 bits
- GPTQ (October 2022) compressed 175B-parameter models to 3 or 4 bits per weight with negligible accuracy loss, running one on a single GPU
- Fine-tune while quantized
- QLoRA (May 2023) fine-tuned a 65B model on one 48 GB GPU by combining 4-bit weights with LoRA
- On your phone
- Apple's on-device model (June 2025): about 3B parameters at 2 bits per weight
Questions people ask
Does quantization make a model worse?
Slightly, and how much depends on the bits. 8-bit is usually close to the original. 4-bit loses a little and is a popular balance. Below about 3 bits, quality often drops noticeably unless the model was trained for it.
What do 4-bit and 8-bit mean?
How many bits store each number in the model. 8 bits allow 256 different values; 4 bits allow 16. Fewer bits means a smaller, faster model and a little more rounding error.
What is GGUF?
A file format for running models locally, made by the creator of the llama.cpp project. GGUF files usually hold quantized models, and their names include codes like Q4_K or Q8_0 that tell you the precision.
What is the difference between quantization and distillation?
Quantization keeps the same model and stores its numbers more coarsely. Distillation trains a separate, smaller model to imitate a bigger one. Both make AI cheaper to run, and are often used together.
How much memory does a model need to run?
A rough rule for the weights: parameters in billions × bits ÷ 8 gives gigabytes. A 7 to 8B model at 4-bit needs about 4 to 5 GB, plus some room for the conversation. Longer context windows need more.
Further reading
- Quantization concept guideHugging Face, 2024
- LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (Dettmers et al.)arXiv, 2022
- GPTQ: Accurate Post-Training Quantization for Generative Pre-trained TransformersarXiv, 2022
- GGUFHugging Face Hub docs, 2026
- Apple Intelligence Foundation Language Models: 2025 updatesApple Machine Learning Research, 2025