Lesson 3 of 7
Assembling a GPT block
8 min read
One attention layer is clever but shallow. How do you turn it into something dozens of layers deep without it falling over?
One block, stacked N times
A Transformer block is a fixed recipe you repeat. Inside: a self-attention layer, then a small feed-forward network that thinks about each token on its own. Around each, two supports — a residual connection that adds the input back to the output, and layer normalization that keeps the numbers in a sane range. Stack the identical block dozens of times and depth does the rest.
Every token stays a vector of numbers the whole way up. Each block nudges those numbers to carry a little more meaning.
Why the supports matter
Residuals and normalization aren't decoration — they're what let you train something this deep. The residual gives the training signal a clean path back through every layer; normalization stops the numbers exploding or vanishing. Skip them and a deep stack simply won't learn. The block's parameters — the weights in attention and the feed-forward net — are what training will tune.
Depth is just the same block repeated. The residual-and-norm scaffolding is what makes stacking it deep actually trainable.
Because every block is identical, you write it once as a class and loop. A model's 'size' is mostly this loop count times the width of each layer.
What you built
- —A block: attention + feed-forward, each wrapped in a residual and normalization.
- —A stack of identical blocks — depth from repetition.
- —A final layer that turns the top vector back into a score per possible next token.
Why do Transformer blocks include residual connections and normalization?
Continue in the app
Take the whole Build an LLM from Scratch course — tracked
Get your personalized path, progress and streaks in the app — this lesson and every next one, in order.