Trace-based production evaluation (AI observability)

How it works

Offline suites only test the inputs you thought of. A traced system records what actually happened on every real request: the assembled prompt, which documents were retrieved and their scores, model and version, each tool call and its result, token counts and cost, latency per span, and where the request ended — answered, refused, errored, escalated to a human. Traces turn production into the largest test corpus the system has: failures are reproducible from the recorded span rather than from a user's paraphrase, evaluators can be run over sampled live traffic instead of a frozen set, and regressions surface as a shift in a distribution you're already collecting. It is also the prerequisite for the other production methods — a canary or an A/B test can only compare what you instrumented.

When to use it

Before any production launch; when debugging failures that only appear with real users; to source evaluation cases and regression tests from real traffic rather than imagination.

Limitations

Traces are raw material, not a verdict — without evaluators and alerting on top, you have storage rather than testing. Prompts and retrieved documents are often personal or confidential, so retention, redaction and access control are part of the design, not an afterthought.

Cite this