Lesson 3 of 6
LLM-as-judge
8 min read
Exact-match works for '2 + 2 = 4.' But how do you auto-grade a summary, a tone, a helpful reply — where a hundred different answers are all fine?
Grade with a rubric, not a string match
For open-ended output you can't diff against one golden string. Instead you hand a second model — the judge — the answer plus a rubric (the criteria a good answer must meet) and ask it to score. Now 'quality' is something you can define, apply consistently, and run at scale.
An LLM judge scores output against criteria you write. The rubric is your definition of good — change it and the winner changes.
The judge is a model too
Don't hand the judge a blank check. It's a model, so it can be swayed by length or confident phrasing, and it can be poorly calibrated — sure of a wrong verdict. Keep the rubric concrete, spot-check the judge against your own labels, and prefer clear yes/no criteria over a vague '1–10 for quality.'
A judge scales your taste; it doesn't replace it. Validate the judge against human labels before you trust its scores.
A vague rubric produces vague scores. 'Is it good?' invites the judge's bias; 'Does it cite the policy? (yes/no)' gives you something you can trust and audit.
The shape of it
- —Use a judge model for open-ended output with no single right answer.
- —The rubric is your definition of quality — concrete, ideally yes/no criteria.
- —The judge is a model: it can be biased or miscalibrated, so validate it.
- —Spot-check judge scores against your own labels before you rely on them.
Your LLM judge rates every answer a confident '9/10,' including ones you know are wrong. What's the fix?
Continue in the app
Take the whole Evals & Reliability course — tracked
Get your personalized path, progress and streaks in the app — this lesson and every next one, in order.