Lesson 3 of 7
Why inference is slow & expensive
7 min read
The model seems to 'know' the answer the instant you hit enter — so why does it type it out slowly, token by token, like it's thinking?
One token at a time
Generating text is a loop. To produce the next token — a word or word-piece — the model runs its full stack of maths over everything so far, picks one token, adds it, and runs the whole thing again for the next. A 500-word answer is hundreds of passes through billions of weights. It's sequential: token N+1 can't start until token N exists. That's why long answers take longer — and cost more.
Inference is a loop: one full pass through the model per token. More tokens out means proportionally more compute — and more time.
The tokens you never see
It gets pricier with reasoning models. A reasoning model first writes a long private scratchpad — reasoning tokens — before its short visible answer. You don't see that thinking, but every hidden token was a full pass through the model, and it's billed at the output rate. Ask a 'thinking' model a simple question and you can pay for hundreds of invisible tokens on top of the ten you read.
A reasoning model spends invisible tokens to think. They cost real compute and real money, even though you never read them.
This is why a bigger or 'thinking' model isn't always the right call. If a task doesn't need deep reasoning, a smaller model that skips the scratchpad is faster and cheaper for the same visible answer.
The shape of it
- —Text is generated one token at a time — each token is a full pass through the model.
- —More output tokens means more compute, more latency, more cost — it scales with length.
- —Reasoning models add hidden thinking tokens, billed at the output rate, on top of what you read.
Two models return the same short answer, but one is a reasoning model. Why might it cost much more?
Continue in the app
Take the whole Inference & Hardware course — tracked
Get your personalized path, progress and streaks in the app — this lesson and every next one, in order.