Lesson 2 of 8
Where AI's knowledge comes from
10 min read
Ask AI about almost anything — history, cooking, the rules of chess — and it has an answer ready. Nobody sat and typed all those facts into it. So where did it actually learn them?
It learned by reading
An AI like this is built by showing it an enormous slice of the internet — books, articles, websites, conversations, code — and having it play one game on all of it: guess the next word (the same game from the last lesson). Hide the next word, let it predict, check, correct, repeat — billions of times over. Nobody hands it a list of facts. But to get good at guessing the next word in a sentence about Paris, it has to soak up what people usually say about Paris. The knowledge sticks as a side effect of getting good at the game.
All that reading gets squeezed down into the model. It doesn't keep the pages — it keeps the patterns in them. That whole process of reading-and-squeezing is called training.
Look inside: millions of tiny knobs
So where do those patterns actually live? Picture the model as a wall of tiny knobs — millions of them, each just a number the training can turn a little. Reading a page nudges a whole bunch of them a hair. Nobody sets them by hand; the reading does. And the model's guess for the next word isn't looked up anywhere — it's a blend of where all the knobs currently sit.
Those knobs have a name: the model's weights, or parameters — the millions of numbers that training adjusts. A fact isn't filed in one spot; it's smeared across all of them at once. That's why the model can know something without having any single page to point to.
This is also why the squeeze is lossy. A fact you meet on a thousand pages shoves the knobs the same way a thousand times, so they end up firmly set — and the answer comes back sharp. A detail you meet once barely moves them, and the next thing you read nudges them back — so it comes out faint, or gone. There simply aren't enough knobs to hold every one-off detail; everything has to share.
There's a flip side. Because facts share the same knobs, the model can quietly blur two half-remembered ones into a single confident-sounding answer that was never true — a first glimpse of why it makes things up, which is the next lesson.
A summary, not a copy
Here's the catch: the finished model is far smaller than everything it read — picture a giant library boiled down to a single pocket summary. And a summary keeps the gist while throwing the fine detail away. So ask about something common, repeated across a thousand pages, and the answer comes back crisp. Ask about a tiny detail that showed up once, in one forgotten corner of the web, and it turns vague — that detail didn't survive the squeeze.
The model is a lossy summary of what it read: common, repeated things come back clear; rare, mentioned-once details come back fuzzy or not at all. Squeezing the whole internet into a small model means some of it is simply gone.
Its knowledge has an end date
And all that reading happened once, up to a certain moment in time. Everything after it — yesterday's headlines, a film released last week, who won the match last night — was never in what the AI read. So it can be rock-solid on the past and completely blank on the present, even on something the rest of us already take for granted.
Drag along the timeline. Past the cutoff line, the AI draws a blank.
The moment its reading stopped is its knowledge cutoff. Ask about anything after that date and the AI simply wasn't there to read about it.
Some assistants can also look things up on the web while they answer — that patches the gap, but it's the AI reaching out for fresh information in the moment, separate from what it actually learned during training.
Recap
- —An AI learns by reading a huge slice of the internet and practising next-word guessing on it
- —All that gets squeezed into a smaller model — a lossy summary that keeps common patterns and loses rare details
- —Its reading stopped at a knowledge cutoff, so it doesn't know what happened after that date
An AI finished its training a year ago. You ask it who won a tournament held last week. What's the most likely outcome?
Continue in the app
Take the whole AI from Scratch course — tracked
Get your personalized path, progress and streaks in the app — this lesson and every next one, in order.