Lesson 2 of 7

Coding attention

9 min read

Change one word at the end of a sentence and 'it' suddenly means something else. How does code make a token read its neighbours?

The one mechanism that matters

Attention is the heart of the whole model, and it's simpler than it sounds. For each token you compute three vectors from its embedding — a query (what am I looking for?), a key (what do I offer?), and a value (what I'll pass on). A token compares its query to every earlier key, and the closer the match, the more of that token's value it pulls in.

A token's meaning isn't fixed — it's mixed together from the earlier tokens it attends to. That mixing is attention.

Self-attention, then many heads

Because each token attends to others in the same sequence, this is self-attention. You run several in parallel — attention heads — so one head can track grammar while another tracks who-did-what. And you mask the future: a token may only look backwards, never at words it's meant to predict.

Multiple heads read different relationships at once; the mask keeps the model honest by hiding what comes next.

In Raschka's code this is a small class: three linear layers for query, key, and value, a scaled dot-product, a causal mask, then stack the heads. That class, repeated, is most of a GPT.