Quantization

Quantization is the technique of storing a trained model's weights — and sometimes its intermediate activations — in a lower-precision numeric format, such as 8-bit or 4-bit integers instead of 16-bit floating point, to shrink the model's memory footprint and increase inference throughput, at a small and measurable cost in output quality.

A model's weights are just numbers, and the format used to store each one is a size-versus-precision trade-off. Full precision (typically FP16 or BF16, 2 bytes per parameter) preserves the most detail but costs the most memory. Quantization asks: how much of that precision can be discarded before the model's actual outputs get noticeably worse — and for most weights, in most models, the answer is quite a lot.

How it works

A calibration process — methods like GPTQ, AWQ, or bitsandbytes are common implementations — maps each weight's original range of floating-point values onto the much smaller set of values a lower-precision format can represent, chosen to minimize the error this introduces. This works well because most values in a trained weight matrix cluster in a narrow range rather than spreading evenly across the full range floating point could represent, so a coarser format loses comparatively little. Going all the way to 8-bit integers is close to free in practice; going to 4-bit and below needs a more careful calibration process to keep quality loss small, since there's simply less room to represent each weight's original value.

Why it matters

Memory is often the actual limit on whether a model can be run at all, separate from whether it's capable enough for the task. A model that needs several high-end GPUs at full precision might run on a single GPU quantized to 8-bit, or become runnable on a laptop at 4-bit where it wasn't runnable at all before. Quantization is the primary lever between "impossible to self-host" and "runs on the hardware I already have," and the quality cost is typically small — often under a couple of percentage points on standard benchmarks at 8-bit, somewhat more at 4-bit — against a multi-times reduction in memory.

A worked example

A 7-billion-parameter model needs 14 GB of VRAM at FP16 (2 bytes per parameter: 7 billion × 2 bytes = 14 GB). Quantized to INT8 (1 byte per parameter), it needs 7 GB — half. Quantized to INT4 (0.5 bytes per parameter), it needs 3.5 GB — a 4x reduction from full precision. That's the difference between requiring a data-center-class GPU and fitting comfortably on a consumer graphics card, for the same model, with a typically small and measurable quality cost rather than a free lunch.

How miatz teaches it

The applied-AI track has learners compute the VRAM math above themselves for several real model sizes before ever running a quantized model, so the trade-off is arithmetic they own rather than a fact they're told. The lab pairs a quantized and full-precision version of the same model against an identical eval set, so learners see the actual quality delta on their own hardware rather than trusting a benchmark number from someone else's setup.