Plain-English explanation
A model's weights are numbers, and those numbers can be stored with more or less numerical precision: think of it like the difference between writing a price as $19.99 versus rounding it to $20. Models are usually trained using high-precision 32-bit or 16-bit floating-point numbers, which is accurate but takes a lot of memory to store and move around. Quantization converts those numbers to a lower-precision format (commonly 8-bit or 4-bit integers), which makes the model file dramatically smaller and faster to run, in exchange for a small amount of accumulated rounding error.
How it works
The most common approach, affine (or "asymmetric") quantization, works by finding the range of values in a tensor of weights, then mapping that range onto the much smaller set of values a lower-precision format can represent: for example, mapping a range of 32-bit floats onto the 256 possible values of an 8-bit integer, using a scale factor (and often a zero-point offset) to convert between the two. Roughly speaking, an 8-bit (int8) model is about four times smaller than its 32-bit float counterpart, and a further step down to 4-bit roughly halves that again, though 4-bit typically introduces a larger accuracy drop and often benefits from more advanced quantization methods (such as GPTQ or AWQ) to hold up quality. Quantization can happen after training is complete (post-training quantization, the common approach for locally-run models) or be simulated during training itself (quantization-aware training) to help the model adapt to the lower precision in advance.
Adult-AI use
Quantization is the main reason a hobbyist can run a capable local language model or image generator on a single consumer GPU (or even a CPU) at all: the full-precision version of a large model often simply won't fit in the amount of VRAM most people have. In practice, this shows up as a choice users make directly: model-sharing pages for local companion chatbots and image generators commonly offer the same underlying model at several quantization levels, and picking one is a real tradeoff between output quality, generation speed, and whether the model fits in memory at all on a given machine.
How it differs from GGUF
Quantization and GGUF are related but not the same thing, and mixing them up is common. Quantization is the technique, reducing numerical precision. GGUF is a file format used by the llama.cpp/GGML ecosystem that commonly stores quantized weights, but a GGUF file isn't required to be quantized (it can store full-precision weights too), and quantization isn't unique to GGUF: plenty of quantized models are distributed as SafeTensors files instead, for use with other tools. A single GGUF-distributed model is typically offered at several different quantization levels (commonly labeled things like Q4_K_M or Q8_0), each trading off file size and speed against quality differently.
Limitations and misconceptions
- Quantization isn't free. It always introduces some approximation error; the resulting quality loss varies by model, method, and use case, which varies by model and quantization level.
- Not all quantization methods are equal. Different techniques (naive rounding vs. more sophisticated calibrated methods) produce meaningfully different quality at the same bit-width.
- Lower isn't always the right choice. The "best" quantization level depends on available hardware and how much quality loss is acceptable: there's no single correct answer.
Technical context
Image and language stacks quantize different components and formats, producing different loader and quality trade-offs.