Lesson 4 of 8
How AI became a helpful assistant
10 min read
A model trained on the internet can predict text brilliantly — but that alone doesn't make it helpful. So how did a raw next-word machine turn into something that actually answers your questions?
Fresh out of reading, it's no assistant
Straight after all that reading, the model can do exactly one thing: continue text in a plausible way (lesson one). That is not the same as being helpful. Ask this raw version a question and it might not answer at all — it might just tack on three more questions, because on the internet, questions so often arrive in lists. It's completing a document, not helping a person. This raw, just-trained version is called a base model.
Reading the internet produces a base model: brilliant at continuing text, but with no notion that it's meant to be a helpful assistant. Knowing a lot isn't the same as being helpful.
Taught good manners by example
So people taught it how to behave — by example. They showed it thousands of model conversations: here's a question, here's a clear and helpful answer; here's a rude or tricky request, here's a calm, honest reply. The model copied the pattern of those examples. It didn't pick up new facts here — it picked up a new manner: answer the question, be useful, stay civil. This second stage is called finetuning.
A second stage, finetuning, shapes how the model behaves by showing it example conversations. The reading gave it knowledge; finetuning gave it the helpful, answer-the-question manner you're used to.
Look inside: it becomes what you reward
The examples teach it a way to answer — but which way is best? That last polish comes from people. The model shows two answers to the same question, and a person picks the better one. Do that thousands of times and the model drifts toward the kind of answer that keeps getting picked. You're not writing its replies; you're rewarding the ones you like, and it leans that way.
This last stage is learning from human feedback — people rating answers, the model tuning itself toward the ratings. In the trade it's called RLHF (reinforcement learning from human feedback). It adds no new facts; it steers the manner, using your preferences as the compass.
It's powerful, and it's also the catch: the model becomes exactly what gets rewarded — not what's actually good. Reward answers that merely sound confident and helpful, and it learns to sound that way, whether or not it's right. Get the reward subtly wrong and you get a model that's polished, agreeable, and quietly off. What you reward is what you get.
This is also why two assistants trained on similar text can still feel so different — friendly, terse, cautious, chatty. Much of that personality is the reward step, not the reading.
So the stages stack up. Pretraining — the reading, where the knowledge comes from. Then finetuning — the examples, where the helpful manner comes from. And a final polish where people rate its answers, good vs. better, to steer it the rest of the way.
Recap
- —Reading the internet produces a base model — it can only continue text, not help you
- —Finetuning on example conversations teaches it to answer helpfully and politely
- —Two stages: pretraining gives it the knowledge, finetuning gives it the helpful manner
- —A final stage — people rating its answers (RLHF) — steers the manner toward what they prefer
An assistant answers your question politely and stays on topic. Which stage taught it to behave that way?
Continue in the app
Take the whole AI from Scratch course — tracked
Get your personalized path, progress and streaks in the app — this lesson and every next one, in order.