Glossary
What is attention (in a transformer)?
How a model works out which earlier words each word should listen to. For every word it looks back over the text, weighs the others by how relevant they are, and pulls in meaning from the ones that matter — that's how it figures out what a word like "it" refers to. It's the core mechanism inside a transformer.
As the model reads, every word needs to know which OTHER words it depends on. In “the cat sat because it was tired,” does “it” mean the cat? Attention is the mechanism that lets each word look back and weigh how much every earlier word matters to it.
How much each word attends to the others
Read a row: “it” looks hardest at “cat” — that's how the model works out what “it” refers to.
Stacked across many layers, attention is what lets a transformer track meaning across a whole context window — the 2017 breakthrough behind every modern language model.
Related terms
Appears in these lessons
See it connect
Watch a word find its reference
How AI Really Works lights up which words attend to which, live — free to try in the app.