Lesson 4 of 7
What "it" means
9 min read
"The cat didn't cross the road because it was too tired." What does it mean — the cat or the road? You know instantly: the cat. But an AI reads one word at a time, in order. How could it possibly tell which earlier word it points back to?
No word stands alone
A word's meaning is rarely settled on its own. Bank by a river is not the bank that holds your money. It points at a different thing in every sentence. To pin down what any word means here, the model has to look at the words around it — but that raises the real question: which of the other words, and how much should each one count?
No word can be understood by itself. Its meaning is finished off by the company it keeps — so the model's job is to work out, for each word, which other words that company is.
Every word looks back and weighs the rest
This is the job of attention. Inside each layer of the transformer from the last lesson, every word looks back over all the earlier words and gives each one a weight — how much it should listen to that word. Then it pulls in meaning from the ones it weighted highly. In the scene below, the word it is doing exactly this: flip the ending of the sentence and watch where its attention goes.
Same word — it — but change one later word and its attention leaps from cat to road. The model is genuinely working out what it refers to, from the context, not guessing the nearest noun.
Every word, not just "it"
"It" was the easy case. Inside the model, every word does this looking-back at once, weighing the words before it. In the scene below, tap any word to make it the one doing the looking — and watch its bars show how much it leans on each earlier word. Notice the very first word has nothing to look back at: you can only lean on words that already came. Flip the ending and "it" swings its attention from one word to another, right in front of you.
A flat row of words becomes a web: every word quietly weighs every earlier one, all at the same time. That web — rebuilt in each layer — is how the model tracks what a word really refers to.
Why this is the whole idea
Attention is the heart of the transformer — it's the part doing the real work. It's what lets a model follow a long, tangled sentence, keep track of who did what to whom, and connect a word to another one far away. Every layer does this weighing again, over and over, and that repetition is how the knot of a real sentence slowly gets untangled.
Attention is how the model turns a flat row of words into a web of relationships — which word belongs with which. It's the single trick that made today's AI possible.
Under the hood, each word actually asks a little question (a query) and every word offers a key; how well a query matches a key sets the weight. A model runs many of these attention patterns side by side — called heads — each watching for a different kind of relationship (one for grammar, one for who-did-what, and so on).
Recap
- —A word's meaning depends on the other words around it — the model has to find which ones
- —Attention lets every word look back and weigh the earlier words by how much they matter to it
- —It's the core of the transformer — how a flat line of words becomes a web of relationships, repeated in every layer
You give an AI: "The cat didn't cross the road because it was too **wide**." To understand what *it* refers to, which earlier word does the model most need to attend to — and why?
Continue in the app
Take the whole How AI Really Works course — tracked
Get your personalized path, progress and streaks in the app — this lesson and every next one, in order.