Lesson 4 of 7
Pretraining on unlabeled text
9 min read
No one labels a trillion words by hand. So where does all the 'learning' in a language model actually come from?
The whole loop is one guess
Pretraining is astonishingly simple to state: show the model some text, have it predict the next token, compare its guess to the word that actually came next, and adjust. The text is its own answer key — that's why it's called unlabeled. Do this over and over and the model's parameters slowly come to encode how language works.
Learning is just: guess the next token, measure the error, nudge every weight a little the right way. Repeat billions of times.
Loss, gradients, repeat
Each step has three moves. A loss number says how wrong the guess was. Backpropagation works out which direction to move each weight to shrink that loss. The optimizer takes the step. Run this loop across huge amounts of text and you get a base model — a raw next-token predictor that knows a lot but doesn't yet follow instructions.
No single weight holds a fact. Knowledge is smeared across millions of weights, all nudged together by the same predict-and-correct loop.
This is the expensive part — weeks on many GPUs. Everything you coded in the earlier lessons exists so this loop has something to tune.
What you built
- —A next-token prediction loop over plain, unlabeled text.
- —A loss that scores the guess, gradients that point downhill, an optimizer that steps.
- —A base model: fluent at predicting text, not yet at following instructions.
Why is pretraining data called 'unlabeled'?
Continue in the app
Take the whole Build an LLM from Scratch course — tracked
Get your personalized path, progress and streaks in the app — this lesson and every next one, in order.