Lesson 6 of 7
Learning what we prefer
11 min read
“Solve this equation” has a right answer you can check. “Write me a warm thank-you note” doesn't. So how do you train a model to be good at the huge range of tasks where there's no answer key to grade against?
When there's no right answer
Reinforcement learning needed a way to check each attempt. That's easy for maths or code — but most of what we ask AI has no such check. Was that email kind enough? Was that summary clear? There's no equation to test against; there's only human judgement. So instead of checking answers automatically, we ask people: shown two of the model's replies, which do you prefer?
For the many tasks with no checkable answer, the yardstick is human preference: people compare the model's replies and say which is better.
Build a judge, then don't overtrust it
You can't have humans rank millions of answers, though — far too slow. So you gather a batch of their comparisons and train a second model, a reward model, to predict what people would prefer. Now you have an automatic judge, and you can improve the assistant with reinforcement learning against it — this whole recipe is RLHF. But the judge is only an impression of human taste, and if you optimise against it too hard, the model starts gaming it: piling on flattery or length the judge rates highly but real people don't. Build the judge below, then watch it get gamed.
A reward model is a stand-in for human taste, not the real thing. Optimise against a stand-in hard enough and you get reward hacking: a high score and a worse answer.
Watch the gap open
The trouble hides in a single gap. The judge's score and what real people actually think agree — but only up to a point. Below, push the model harder and harder against the judge. Watch the judge's score keep climbing while real people, past a certain point, start to sour. That widening gap between the two lines is the failure. Then refresh the human check and watch it snap shut — until you push past it again.
The judge and real people only agree where humans recently checked. Push past that and they split: the score says “better” while people say “worse.” Keeping the human check fresh is what holds the two together.
Why every stand-in can be gamed
This is closer to a law than a bug. The moment you replace a goal with a measure of that goal, pushing hard enough on the measure drags it away from the goal itself. The judge is a measure of human taste; optimise it too hard and you're perfecting the measure, not the taste. Labs hold the line by not over-pushing, refreshing the human data often, and mixing in problems that can be checked exactly.
This is why assistants can drift into sounding over-eager or sycophantic — telling you what scores well rather than what's true or useful. Labs fight it by not over-optimising, refreshing the human data, and mixing in checkable tasks. It's the central tension of RLHF: the reward is a proxy, and every proxy can be gamed.
Recap
- —Most tasks have no checkable answer, so training uses human preference: people pick the better of two replies
- —A reward model learns to predict those preferences, giving an automatic judge to improve against — the recipe is RLHF
- —The judge is only a proxy for human taste; over-optimising it causes reward hacking — high score, worse answer
- —The judge only matches real people where humans recently checked — push past that and the two split apart
A team trains hard against their reward model and its scores keep climbing — yet testers say the answers are getting worse: longer, gushier, less honest. What's happening?
Continue in the app
Take the whole How AI Learns to Think course — tracked
Get your personalized path, progress and streaks in the app — this lesson and every next one, in order.