Lesson 5 of 7
Quantization
7 min read
A big open model needs more memory than most machines have. So how do people run it on a single gaming GPU — or even a laptop?
Round the numbers down
A model is millions of parameters — numbers — stored at some precision, often 16 bits each. Quantization stores them more coarsely: 8 bits, 4 bits, sometimes fewer. It's like rounding. Halve the bits per weight and you roughly halve the memory the model needs, and speed it up too, because there's less data to move. The catch: round too hard and the numbers get so coarse the model's quality slips.
Quantization trades precision for size and speed. Fewer bits per weight means a smaller, faster model — the numbers are just stored more roughly.
A cheap win, up to a point
The surprise is how little quality you lose at first. Going from 16-bit to 8-bit is often nearly free; 4-bit is where most people land for running big models locally — a small, usually acceptable dip. Push to very low bits and the drop becomes real. So quantization is the standard trick for fitting a model onto smaller hardware: take the biggest model you can, then quantize until it fits your memory without hurting quality too much.
The quality curve is flat, then falls: 8-bit is nearly lossless, 4-bit is a mild trade, and very low bits break things. Quantize as far as the task tolerates — no further.
Quantization changes how the weights are stored, not what the model learned. It isn't the same as a smaller model trained from scratch — it's the same model, compressed. Test quality after quantizing; how far you can drop depends on the model and the task.
The shape of it
- —Weights are stored at some precision (often 16 bits); quantization uses fewer.
- —Fewer bits means less memory and more speed — how a big model fits on small hardware.
- —Quality is nearly untouched at 8-bit, a mild trade at 4-bit, and breaks if you go too low.
You want to run a large open model on a single consumer GPU it won't otherwise fit on. What's the standard move?
Continue in the app
Take the whole Inference & Hardware course — tracked
Get your personalized path, progress and streaks in the app — this lesson and every next one, in order.