Lesson 4 of 6
Regression & CI for prompts
7 min read
You tweak one line of the prompt to fix a bug. It works. What you don't see: the same tweak just broke three cases that used to pass.
Prompts are code
A prompt is a dependency your product's behavior rests on, just like a function. So treat it like one: put it in version control, review changes, and — the part everyone skips — run your evals on every change. A one-word edit can ripple in ways no human will spot by reading the diff.
Every prompt or model change is a code change. If it isn't tested, you're shipping blind.
Wire evals into CI
Make the eval suite a check in your pipeline. Open a pull request that touches the prompt, model, or retrieval, and CI runs the test set and posts the score. If it drops below your baseline, the merge is blocked — the same way a failing unit test stops a bad commit. Silent regressions become loud, before users meet them.
A regression caught in CI costs a code-review comment. The same regression caught in production costs a customer.
Pin the model version. 'The provider updated the model' is a real regression source — if you can't pin it, run your evals on a schedule so a drift shows up as a failing check, not a support ticket.
The shape of it
- —Version your prompts and review changes like any other code.
- —Run the eval suite on every prompt, model, or retrieval change.
- —Block the merge when the score drops below your baseline.
- —Pin the model version — or schedule evals to catch provider drift.
A prompt change fixes the bug you were chasing and the demo still looks fine. Before merging, what does an eval-in-CI setup do?
Continue in the app
Take the whole Evals & Reliability course — tracked
Get your personalized path, progress and streaks in the app — this lesson and every next one, in order.