What is quantization (in plain English)?
Plain-English definition · Updated 2026-10-26. Bit widths below are the GGUF / llama.cpp K-quant conventions current in 2026 — they move with tooling generations, so check the file you actually download.
AITop is an independent guide. Prices and features change fast — always check the vendor's page.
Quantization is how a model is squeezed into less memory: instead of storing every weight at the 16-bit precision used in training (FP16, 2 bytes each), the file is written at fewer bits per weight. That single dial decides whether a model fits your GPU at all. The only formula you need is VRAM ≈ parameters × bytes per weight: a 7B model at FP16 is about 14 GB, while at Q4_K_M (~4.5 bits/weight) it is about 4 GB; a 70B model drops from 140 GB at FP16 to roughly 39 GB at Q4_K_M — the difference between "needs a datacenter" and "fits one 48 GB card". Q4_K_M is the community default because it keeps most of the quality for a quarter of the size; go lower and the quality loss climbs. The catch buyers miss: the KV cache stays full precision no matter how aggressively you quantize the weights, so long-context use still needs the memory.
What it actually means
A model's parameters (its "weights") are just numbers, and each number is stored using some number of bits. Training and the highest-fidelity serving use FP16 or BF16 — 16 bits, or 2 bytes, per weight — which is the reference point everyone quotes sizes against. Quantization stores those same weights at 8 bits, 4 bits, or even less. Fewer bits per weight means a smaller file, less VRAM to hold it, and — because every token generated has to be shuttled through memory — often faster generation on the same hardware.
The trade is accuracy. Squashing 16-bit numbers into 4 bits throws away information, and the model gets slightly worse at its job. The art is doing it where the eye (or the benchmark) won't notice. Modern formats like llama.cpp's K-quants are not uniform: a "Q4" file mixes ~4-bit storage for most weights with ~6-bit storage for the layers that are most sensitive to precision loss, which is why the real average lands at ~4.5 bits per weight rather than a clean 4. The label on the filename — Q2_K, Q3_K_M, Q4_K_M, Q5_K_M, Q8_0 — is the headline you actually shop by.
Why it matters when you're picking a tool
Because it is the difference between "this runs on my machine" and "this needs a cloud quota." The math is one line: VRAM for weights ≈ parameters × bytes per weight. Worked examples that match every calculator and vendor card:
FP16 (16 bits, 2 bytes/weight) — a 7B model is ~14 GB, a 70B model is ~140 GB. This is the unquantized reference; almost nobody runs it locally. Q8_0 (8 bits, ~1 byte/weight) — roughly half the size, quality loss usually described as near-imperceptible (a percent or two on perplexity). Q4_K_M (~4.5 bits/weight) — about a quarter of the FP16 size; a 70B model lands near 39 GB, which is why a single 48 GB workstation card can serve it. Q3_K_M (~3.5 bits/weight) and Q2_K (~2.5) save more still but the quality loss becomes noticeable on harder tasks.
Two buying rules fall out of this. First, bigger models quantize better: a 70B model at Q4_K_M loses proportionally less quality than a 7B model at the same precision, because there is more redundancy to spare — so if you have the VRAM for a bigger quantized model, take it over a smaller model at higher precision. Second, mixture-of-experts models quantize especially well, because only a subset of parameters is active for any given token, so a coarse weight format hurts less (see mixture of experts). Q4_K_M is the default recommendation for exactly this reason: for most local users it is 90%+ of the quality at 25% of the memory.
The 2026 reality check
The precision ladder is now a hardware feature, not just a file trick. NVIDIA's Blackwell generation added native FP4 (NVFP4) support, making 4-bit a compute-accelerated precision that GPUs can actually multiply in, not merely a storage format — which is why "4-bit" stopped meaning "slow and lossy" in 2025–2026. FP8 is likewise a first-class inference precision now, sitting between INT8 and FP16. The practical buyer still lives in the GGUF K-quant world for local models, but the underlying bit widths are no longer exotic.
The bits-per-weight number on the filename is the only thing you can trust at a glance. Quality loss is measured with perplexity (how hesitant the model is on a fixed test text), and it is not a human star rating — vendors who say "no quality loss" are selling, not measuring. The widely repeated community range is ~5–10% perplexity loss for Q4_K_M and ~1–2% for Q8_0, but the real number depends on the model, the task, and the temperature, so treat those as orders of magnitude, not specs. One phrase to distrust on sight: "2× faster" from quantization alone, quoted without saying it assumes INT4/FP4 hardware — on a CPU or an older GPU the win is the memory saving, not a flat 2×.
Quantizing the weights does not quantize the memory that grows with your conversation. The attention state (the KV cache) is normally kept at FP16 regardless of how aggressively you compressed the weights, so a model that fits at Q4 with a short context can still balloon past your VRAM at a 32K or 128K context window — that budget scales with context length and architecture, not with parameter count (see context window). "Fits in 4 GB" almost always means "fits at a short context."
Quick checklist
- Memorize one formula: VRAM ≈ parameters × bytes per weight. 16 bits = 2 bytes (FP16), 8 bits ≈ 1 byte (Q8_0), ~4.5 bits ≈ 0.56 byte (Q4_K_M). A 7B/70B model is 14/140 GB at FP16 and ~4/39 GB at Q4_K_M — do the multiplication before you buy the card.
- Default to Q4_K_M unless you have a reason not to. It is the community standard for a reason: most of the quality at a quarter of the size. Step up to Q5_K_M or Q8_0 only when you have the VRAM and the task is precision-sensitive (code, exact math).
- Bigger quantized beats smaller precise. A 70B model at Q4_K_M usually beats a 14B model at Q8_0 on hard tasks, because larger models tolerate lower precision better. Spend your VRAM on parameter count first.
- Read the bits-per-weight, not the marketing. "Runs locally" with no quant named usually means Q4 or lower; check the filename. And ignore "no quality loss" and bare "2× faster" claims that don't state the hardware and the perplexity they measured against.
- Budget the KV cache separately. Weights at Q4, but the context cache stays FP16 — a model that fits at 4K context can overflow the same card at 32K. The context length, not the weight format, often decides the real ceiling.
Where you'll hit it
Quantization is the quiet variable in three purchases you will meet on this site: the Ollama vs LM Studio comparison (both are local runners, and which quant you pull is the entire "will it fit?" question), the best AI chatbots ranking (where model size and context length decide whether a tool is usable on your hardware), and mixture of experts (the architecture that quantizes best). Related terms: context window is the budget the KV cache shares with your text, and inference cost is what the bits-per-weight decision does to your per-token bill. Full list on the AI tools glossary.
The bottom line
Quantization is just the bits-per-weight dial, and the only number that matters to a buyer is one formula: VRAM ≈ parameters × bytes per weight. At FP16 every weight is 2 bytes, so a 70B model needs 140 GB; at Q4_K_M (~4.5 bits) it needs about 39 GB — the gap between a datacenter and a single 48 GB card. Q4_K_M is the default because it keeps most of the quality for a quarter of the size, bigger models survive lower precision better than small ones, and mixture-of-experts models survive it best of all. The things to keep honest: the KV cache stays full precision no matter how you compress the weights (so long context still blows the budget), "no quality loss" and bare "2× faster" are marketing unless they name the hardware and the perplexity, and the bit width on the filename is the only spec you can trust at a glance. Read the quant, do the multiplication, then buy the card.