Lesson 1 of 7
The internet, compressed
10 min read
You could never fit the internet on your phone. Yet an AI that read a huge slice of it fits in a file you could download overnight. Almost everything it read is gone — so how is what's left good enough to answer you?
An impossible file
Think about how much text is on the internet: billions of pages — more than anyone could read in a thousand lifetimes. Now think about the AI that learned from it. Its finished form — the model — is astonishingly small next to all that: not a warehouse of pages, but a compact set of patterns. It plainly can't be holding a copy of everything it read. There isn't room.
The model isn't a folder full of web pages. It's a squeezed-down version of what was in them — the shape of the information, not the pages themselves.
One game, a trillion rounds
How do you squeeze a slice of the internet into something that small? You don't hand the AI a list of facts. You make it play one tiny game, over and over: cover the next word in some real text and make it guess. Show it "The capital of France is ___" — it guesses; you reveal "Paris" — it nudges itself to be a little less wrong. Repeat across all that text, billions of times. That endless guess-check-correct is the whole of training.
To keep winning that game everywhere, memorising isn't an option — there's nowhere to put it all. The only way through is to find the patterns that generate the text, and keep those instead.
What survives the squeeze
So the model keeps patterns, not pages — and a pattern is a lossy thing. Something the internet repeats on a thousand pages leaves a deep, sharp groove. Something mentioned once, in a forgotten corner, barely leaves a trace. Train the AI below, then ask it two things: one common, one obscure. Watch which one comes back clear.
Same small model, two very different answers: the common fact reconstructs crisp, the rare detail comes back blurry or missing. It didn't store either — it rebuilt them from patterns, and only the common one left a strong enough pattern to rebuild.
Squeeze it yourself
The scene above trained a model for you. Now take the squeeze into your own hands. Below are four facts the model read — some seen on millions of pages, some just once. Drag the model smaller and watch the rare facts blur and drop out first, while the common ones stay sharp. Then tap a fact to make it appear more often, and watch a lost one come back clear. Same squeeze — but what you feed it more of is what it keeps.
Nothing here is stored, only rebuilt. Shrink the model and the bar a fact must clear to survive rises, so the rarest ones vanish first. It isn't the size of a fact that saves it — it's how often it was seen.
To compress is to understand
Here's the part that matters. To guess the next word of a sentence about gravity, the AI has to have absorbed a little of how gravity works. To finish a line of a story, it has to track who wants what. Good compression isn't just shorter — it means finding the rules underneath, because rules are what let you regenerate the details you threw away. Squeezing the internet well and understanding it turn out to be almost the same task.
That's the quiet engine behind everything AI does: pushed to predict the next word with almost no room to cheat, the model is forced to build a rough working model of the world that produced the words.
Under the hood, this reading-and-squeezing stage has a name — pretraining — and what it produces is a base model. The rest of this course opens up how the guessing actually happens: how your words become numbers, what the machine inside does with them, and how it turns all that back into the next word.
Recap
- —An AI can't store the internet — its model is far too small, so training compresses it
- —That compression is one game repeated endlessly: guess the next word, check, correct
- —It keeps patterns, not pages — common things survive sharp, rare details blur away — and finding those patterns is a rough form of understanding
Two facts sit online: one repeated across millions of pages, the other written once in a single forum post from years ago. After training, which is the AI more likely to recall accurately — and why?
Continue in the app
Take the whole How AI Really Works course — tracked
Get your personalized path, progress and streaks in the app — this lesson and every next one, in order.