Lesson 4 of 6

Distillation

6 min read

A frontier model gives great answers but costs a fortune and runs slowly. What if a small model could learn to imitate it?

The big model as a teacher

Distillation turns an expensive model into a cheap one. The large teacher model labels a pile of examples — questions with its best answers — and a small student model is fine-tuned on those labels. The student learns to reproduce the teacher's behavior, batch by batch, until it lands near the teacher's quality.

The student never sees a human answer key — it learns from the teacher's outputs. That's the trick: one slow, expensive model creates the training data for a fast, cheap one.

Most of the quality, a fraction of the cost

A distilled student rarely matches the teacher exactly — it settles near it. But it runs far faster and cheaper, which is often the whole point: you can serve it at scale, on-device, or under a tight latency budget. Many of the small models you use every day are distilled from a larger sibling.

Distillation trades a small quality drop for a big cost-and-speed win. Serving millions of calls, near-teacher quality at a tenth of the cost is the better deal.

Distillation is one more reason to reach for fine-tuning wisely: sometimes the win isn't a smarter model, it's the same behavior in a cheaper one.