Lesson 5 of 7
Rolling the dice
10 min read
You type "The weather today is" and hit go. The AI could write sunny, freezing, glorious — even a whole poem. It doesn't know which word is right. So how does it land on one? It plays the odds.
Not one word — odds on every word
Last lesson, the top of the transformer turned your text into "a guess for the next word." Here's what that guess actually is. The model doesn't choose a word — it scores every token in its vocabulary and turns those scores into probabilities. For "The weather today is" it might land on sunny 40%, cloudy 22%, cold 15%, and a long tail of thousands more. Its real output is this whole ranked list of odds.
The model's true answer isn't a word — it's a probability for every possible next word at once. Committing to a single one is a separate step.
Then it rolls the dice
To actually write, it has to commit to one word — so it samples from those odds, like drawing from a weighted bag. A likely word usually wins, but not always. Then it adds that word to your text and runs the whole machine again for the next word, and the next — building the reply one token at a time.
A sentence isn't planned in advance. It's produced token by token, each one sampled fresh from the odds, then fed back in to shape the word that follows.
The creativity knob
How boldly it draws from the bag is a single setting: temperature. Turn it down and the model almost always takes its top pick — steady, predictable, a little dull. Turn it up and unlikely words get a real chance — surprising, creative, sometimes off the rails. The odds underneath don't change; temperature just reshapes how sharply the model favours the front-runner.
Temperature doesn't teach the model anything new — it only tunes how much it plays it safe versus takes a risk when it rolls the dice.
Under the hood, turning the raw scores into probabilities is a step called softmax; committing to one word is sampling; and feeding each chosen token back in to pick the next is what autoregressive means. It's the same model run again and again — once per word.
Recap
- —The model's output isn't a word — it's a probability for every possible next token
- —It writes by sampling one word from those odds, then repeating for the next — token by token
- —Temperature tunes the gamble: low plays safe on the top word, high lets unlikely words through
You ask an AI the exact same question twice and get two different answers, word for word. What's the most likely reason?
Continue in the app
Take the whole How AI Really Works course — tracked
Get your personalized path, progress and streaks in the app — this lesson and every next one, in order.