Lesson 5 of 7

Scale: data, params, compute

8 min read

Everything's coded and it trains. Why does a bigger version of the exact same code suddenly do things the small one never could?

Bigger, on two axes

The code doesn't change when you scale — the numbers do. Model size (parameters) and training data grow together, and the compute bill grows with them. Push both and new abilities switch on that were never programmed in — spelling, then arithmetic, then reasoning appear as the model gets big enough and sees enough text.

Skills you never coded emerge from scale alone. But size and data are two knobs, not one.

Balance beats brute force

Scaling laws are the punchline: error falls in a smooth, predictable way as you grow size, data, and compute in proportion. A giant model fed too little text is no better than a small one — and the reverse is true too. The engineering skill is spending your compute budget where it buys the most, not just making one number huge.

Scale is balanced growth. A model twice as big needs more data too, or the extra size just sits idle.

This is why teams estimate a compute budget first, then pick the model size and dataset that fit it — the laws let you predict the payoff before you spend.