Lesson 5 of 6
Evaluating a tuned model
6 min read
Your fine-tuned model aces every prompt in the demo. Ship it? Not yet — those were the prompts you trained on.
The demo is not the evidence
Fine-tuning can make a model brilliant on examples like its training data and quietly worse on everything else — that's overfitting. A cherry-picked demo hides it. The only honest test is a held-out eval set: real cases the model never saw in training, scored against what you actually care about.
Impressive on training-like prompts proves nothing. Judge a tuned model on held-out data, not on the examples it learned from.
Always compare to the baseline
Never evaluate a tuned model in isolation — compare it head-to-head with the model you started from. Sometimes fine-tuning helps your target task but degrades general ability; sometimes a good prompt on the base model would have matched it for free. Track a single metric you can regress against, and re-run it on every change so a silent regression can't sneak through.
The question isn't 'is the tuned model good?' It's 'is it better than the base model on the tasks I care about?' — measured on data it hasn't seen.
Watch for regressions on tasks you didn't tune for. Gaining your target behavior while losing general competence is a common and easy-to-miss failure.
The shape of it
- —Evaluate on a held-out set, never on training-like demos.
- —Compare head-to-head against the base model, not in isolation.
- —Re-run one fixed metric on every change to catch silent regressions.
Your tuned model nails the demo prompts. What's the honest next step before shipping?
Continue in the app
Take the whole Fine-tuning & Customization course — tracked
Get your personalized path, progress and streaks in the app — this lesson and every next one, in order.