INT8 vs INT4 Quantization
Both shrink a model's weights from 16-bit floats to lower-precision integers to cut memory and speed inference — INT8 keeps more precision, INT4 keeps less and shrinks the model further.
| Dimension | INT8 Quantization | INT4 Quantization |
|---|---|---|
| Memory footprint | Roughly half the size of FP16 — a 7-billion-parameter model drops from about 14 GB to about 7 GB. | Wins here Roughly a quarter the size of FP16 — that same 7-billion-parameter model drops to about 3.5 GB. |
| Quality preservation | Wins here Calibration is close to free in practice, typically under a couple of points of benchmark degradation. | Needs more careful calibration (methods like GPTQ or AWQ) and usually gives up more quality than INT8. |
| Hardware accessibility | Fits many models onto a single high-end GPU where FP16 needed several. | Wins here Can put a model within reach of consumer GPUs, or a laptop, where INT8 still would not fit. |
| Inference throughput | Faster than FP16, though gains depend on hardware kernel support for the format. | Wins here Generally faster still on memory-bandwidth-bound serving, though not guaranteed to double again over INT8. |
| Calibration complexity and tooling maturity | Wins here A simpler, more broadly supported quantization path with fewer failure modes. | Needs a more deliberate calibration process to keep quality loss manageable at this precision. |
| Risk on sensitive or edge-case tasks | Wins here Smaller precision loss makes it the safer default when you cannot extensively re-verify quality yourself. | Larger precision loss means edge cases are more likely to be where quality actually degrades. |
| Typical production choice | The right call when quality headroom is thin and hardware is not the binding constraint. | The right call when memory or hardware access is the binding constraint and you have validated quality against your own eval set. |
When to choose INT8 Quantization
Choose INT8 when quality matters most, you have the hardware for it, and you haven't run your own quality eval to validate a more aggressive quantization.
When to choose INT4 Quantization
Choose INT4 when memory or hardware access is the binding constraint — self-hosting on limited GPUs, edge deployment — and you've validated quality on your own eval set.
The verdict
Quantize to INT8 almost by default — the quality cost is minor and the memory savings are real. Go to INT4 deliberately, and only after measuring your own model's quality delta on a real eval set, since the additional loss is real even where it is often acceptable.