FP16 vs INT8 vs INT4
Read firstHow an LLM works
A 70 billion parameter model in 16 bit floats is 140 GB of weights. The largest common data centre GPU has 80 GB. So before it answers a single question, the model needs two GPUs, plus room for the KV cache. Store the same weights in 4 bits and they are about 35 GB: one GPU with room to spare, or a Mac Studio, and each token comes out faster as well.
That is quantization: representing each weight with fewer bits, accepting a small loss in accuracy for a large saving in memory and time. It is why open models run on laptops, why serving costs fell so fast, and it is a standard question for anyone designing inference.
You must serve a 70B model and have one 80 GB GPU per replica. FP16 does not fit. Would you pick INT8 or INT4, and what would you check before shipping it?
Why fewer bits makes it faster, not only smaller#
Generating a token at small batch sizes is not limited by arithmetic. It is limited by memory bandwidth: for every new token, the GPU reads every weight of the model once. A 140 GB model on a GPU that reads 3.35 TB a second (an H100) can produce at most about 24 tokens a second for one user, however fast its maths units are. Halve the bytes and that ceiling doubles.
Full 16 bit precision. The quality reference, and the most expensive way to serve it.
The ceiling is per request. With many requests batched together, one read of the weights serves all of them, and the bottleneck moves towards arithmetic and KV cache memory. That is why quantization helps most for low-batch serving and local models, and why at high batch the format of the arithmetic starts to matter too.