Lesson 4 of 7

Distilled & quantized

6 min read

The best open models are huge — far bigger than a laptop's memory. So how do people run them at home?

Shrink it to fit

A model's real cost is memory: it has to fit in your machine's RAM to run. Full-size models are too big for most laptops. [Quantization](glossary://quantization) shrinks one by storing its numbers more coarsely — think of it as rounding — cutting the size two- to four-fold with only a small drop in quality. A related trick, [distillation](glossary://distillation), trains a small model to copy a big one. Either way: smaller, faster, a little less sharp.

Quantization stores a model more coarsely so it shrinks to fit your machine — for a small quality cost.

Smaller, faster, good enough

Compression is why local AI is practical. A large model that would never fit can drop to a quarter of its size and slip onto an ordinary laptop, running noticeably faster too. You lose a little precision, but for everyday chat and drafting it's rarely noticeable. Runner apps download these compressed versions by default, so most of the time it just works.

A compressed model fits and runs faster; the quality loss is small enough to ignore for everyday work.

If a model runs out of memory or crawls, reach for a smaller size or a heavier compression before giving up — the same model in a lighter form often runs fine.