Lesson 2 of 7
Words become tokens
8 min read
Ask an AI how many R's are in "strawberry" and it may well say two. It can write you a sonnet in seconds, yet miscounts the letters in a word a child can spell. That mistake isn't random — it's a clue to how the AI actually reads.
It doesn't read letters
Before an AI predicts a single word, your text gets chopped into pieces called tokens. It never sees "s-t-r-a-w-b-e-r-r-y" the way you do. Its alphabet isn't the 26 letters — it's a fixed vocabulary of tens of thousands of chunks, and it only ever reads and writes in those chunks.
The model's smallest unit isn't a letter, or even always a word — it's a token: a chunk of text. Everything it reads, it reads as a string of these chunks.
Words become chunks
How is a word chunked? Common words are so frequent they get a token all to themselves — cat is one chunk. Rarer or longer words don't; they're built from smaller pieces the model has seen often — strawberry comes apart into straw + berry. Tap through the words below and watch the same text look one way to you and another to the AI.
To the model, strawberry is just two lumps, straw and berry. It never sees the individual letters tucked inside — so the three R's simply aren't there to count.
Why this explains the weird stuff
Once you know it reads in chunks, a whole cluster of quirks makes sense. It fumbles letter-level tasks — counting letters, spelling backwards, tidy rhymes and anagrams — because those need the letters it never sees. Rare words and typos, which shatter into many odd chunks, throw it off more than common ones. Even cost works this way: text that splits into more tokens takes more work to read and write.
The token is the model's atom. A surprising number of its blind spots aren't about intelligence at all — they trace straight back to the fact that it reasons over chunks, not characters.
Under the hood, the chunks are worked out by a method called byte-pair encoding: it starts from single characters and repeatedly merges the most frequent neighbouring pair into a new token, so common strings end up whole and rare ones stay in pieces. Each model family has its own tokenizer, so the exact split can differ.
The hidden cost of chunks
There's one more place the chunks quietly show up: the bill and the clock. Everything an AI reads and writes is counted in tokens, not words — so text that splits into more chunks costs more to process and takes longer to answer. English packs neatly, a chunk or two per word. But other languages, code, unusual names, or lots of emoji shatter into far more pieces, so the exact same message can cost several times as much. It's why a reply in one language can feel slower or pricier than the same reply in another.
The token isn't only how the model reads — it's the unit it's billed and timed by. Fewer, cleaner chunks mean cheaper, faster answers; text that fragments into many chunks quietly costs more.
Recap
- —An AI reads text as tokens — chunks — never as individual letters
- —Common words are a single chunk; rarer or longer words split into several pieces
- —Because it never sees the letters inside a chunk, letter-level tasks (counting, spelling, rhyme) are exactly where it stumbles
Someone asks an AI to spell "lemonade" backwards, letter by letter, and it botches it — even though it writes flawless paragraphs. Why is this particular task hard for it?
Continue in the app
Take the whole How AI Really Works course — tracked
Get your personalized path, progress and streaks in the app — this lesson and every next one, in order.