Lesson 3 of 8
Why AI makes things up
9 min read
Sometimes AI hands you a fact that's just… wrong — a book that doesn't exist, a quote nobody ever said — and it sounds completely sure of itself. Why does that happen, and why so confidently?
It would rather guess than stop
Remember the AI's one move: predict a plausible next word, again and again (lesson one). When it genuinely knows the answer, the most plausible next words are the truth. But when it doesn't — when that detail got lost in the squeeze (lesson two) — it doesn't stop, and it doesn't feel a gap. It just reaches for the next most plausible-looking words anyway. To a next-word guesser, a made-up book title reads exactly like a real one — so out it comes, every bit as smooth and natural.
When an AI states something false as if it were true, that's a hallucination. It isn't lying — it has no idea it's wrong. It's filling a gap with a plausible guess, because guessing the next word is the only thing it ever does.
Look inside: it grades how it sounds, not whether it's true
Why does a made-up answer come out just as smooth as a real one? Look at how the model chooses. For every answer it could give, it grades one thing: how well this sounds right — how naturally it fits what came before. It has no second grade for whether the answer is actually true. It picks the highest-sounding one and says it. Truth was never on the scorecard.
The model ranks answers by plausibility — how likely the words are — and has no separate signal for truth. When the true words are also the most plausible, you get a right answer for free. When they aren't, it confidently says the plausible thing instead. Same machine, either way.
That's the whole trick behind a confident wrong answer. On a well-worn fact, the truest words are the ones that sound right, so it lands them. On something it never really read, the words that sound right are just… words that sound right — and out they come, fluent and false. The only way to make it prefer the truth is to make the truth the most plausible thing in front of it — which is exactly what giving it a source, or your own notes, does.
Sounding sure isn't being right
And it always sounds sure. That calm, fluent confidence is a style it picked up from millions of well-written pages — and the style gets painted onto every answer, true or false alike. A wild guess comes out in the very same polished voice as a rock-solid fact. So the tone you hear tells you nothing about whether the content is correct.
An AI's confident tone is not evidence. It sounds exactly as certain when it's guessing as when it's right — so you can't judge truth by how sure it sounds.
When it matters, treat specifics — names, numbers, quotes, links, dates — as guesses to check, not facts. Asking "how sure are you, and what's your source?" helps the shaky ones stand out.
Recap
- —An AI always predicts a plausible next word — so when it doesn't know, it fills the gap with a guess instead of stopping
- —A confident, fluent tone is just its default style, painted onto right and wrong answers alike
- —That made-up-but-plausible statement is a hallucination — so verify the specifics that matter
An AI gives you a precise quote, attributes it to a famous author, and sounds completely certain. Smartest next step?
Continue in the app
Take the whole AI from Scratch course — tracked
Get your personalized path, progress and streaks in the app — this lesson and every next one, in order.