Lesson 1 of 7
Text → tokens → embeddings
8 min read
Your model can't read the letter 'a'. So what's the very first line of code standing between a raw string and a neural network?
Cut the text into tokens
Before a single matrix multiply, your code runs a tokenizer: it chops the raw string into small chunks called tokens and maps each to an integer id. A common word is one token; a rare one splits into several pieces. The model only ever sees these ids — never the letters.
Tokenization is step zero. Your whole pipeline runs on token ids, not text — everything after this is numbers.
Turn ids into vectors
An id like 2043 means nothing on its own, so the next layer is an embedding table: one row of learnable numbers per token id. Look up the row and each token becomes a vector — a list of numbers the network can add, scale, and compare. You also add a positional vector so the model knows the order the tokens came in.
An embedding is just a lookup: id → a row of numbers. Those numbers start random and get better as the model trains.
In code this is two tiny steps: build the vocabulary once, then look up embedding[token_ids]. The 'magic' is that the rows are learned, not written by hand.
What you built
- —A tokenizer that turns text into a sequence of integer token ids.
- —An embedding table that maps each id to a learnable vector.
- —A positional signal so word order isn't lost.
Why does the model work with token ids and embeddings instead of the raw text?
Continue in the app
Take the whole Build an LLM from Scratch course — tracked
Get your personalized path, progress and streaks in the app — this lesson and every next one, in order.