Lesson 5 of 6
Observability & tracing
7 min read
A user says the answer was wrong. You open the logs and see... the final reply, and nothing else. Now what?
See inside one request
In production your system is a pipeline — build the prompt, retrieve context, call the model, maybe call a tool. When something goes wrong, the final output alone can't tell you where. Tracing logs every hop of a single request so you can follow it end to end and pinpoint the step that failed.
A trace records each step of one request: the prompt sent, the context retrieved, the model's raw output. When it fails, you can see which step went wrong.
Production is your best eval source
Traces aren't just for debugging — they're a goldmine of eval cases. Every real failure you find in a trace is a case that belongs in your test set. Log the inputs, sample the traffic, watch for the weird ones, and turn each genuine failure into a new test. Your eval set grows from reality instead of your imagination.
Every production failure is a free eval case. Trace it, understand it, then add it to the test set so it can never regress silently again.
Log responsibly. Traces capture real user input, so redact secrets and personal data before they land in your logging system — an observability pipeline is not a place to leak PII.
The shape of it
- —Trace every step of a request so a failure points at the exact hop.
- —The final output alone can't tell you where a pipeline broke.
- —Mine real failures from traces and add them to your test set.
- —Redact secrets and personal data before you log it.
You find a genuine failure while reading a production trace. Beyond fixing it, what's the highest-leverage move?
Continue in the app
Take the whole Evals & Reliability course — tracked
Get your personalized path, progress and streaks in the app — this lesson and every next one, in order.