Lesson 3 of 6
Preparing the data
6 min read
Two teams fine-tune the same model. One uses 5,000 scraped examples, the other 300 hand-checked ones. The smaller set wins. Why?
Garbage in, tuned-in garbage out
A fine-tuned model imitates its examples faithfully — including their mistakes. If your data is inconsistent, mislabeled, or off-tone, the model learns that. This is why a few hundred clean, consistent examples usually beat thousands of noisy ones: you're not teaching facts, you're teaching a pattern, and every bad example blurs it.
Your dataset is the spec. The model copies what you show it, so consistency in your examples becomes consistency in its behavior.
What good data looks like
Good fine-tuning data is consistent in format, representative of the real inputs you'll see in production, and diverse enough to cover the edge cases. Match the exact prompt structure you'll use at inference. Hold out a slice you never train on, so you can measure honestly later. And keep humans reviewing labels — label quality is the ceiling on your model's quality.
Never fine-tune on customer data without checking consent and privacy rules. Training data can be memorized and later surfaced — treat the dataset as if it could leak.
The shape of it
- —Quality and consistency beat raw volume — a few hundred clean pairs go far.
- —Match the example format to your real inference-time prompts.
- —Hold out a test slice you never train on, for honest evaluation.
- —Review labels and check privacy before anything goes into training.
You have 4,000 examples, but they were labeled by different people in inconsistent formats. Best move before fine-tuning?
Continue in the app
Take the whole Fine-tuning & Customization course — tracked
Get your personalized path, progress and streaks in the app — this lesson and every next one, in order.