Application evals — testing AI systems where the model is one component of a larger piece of software. Each entry is a repeatable technique for catching regressions, measuring accuracy, and finding the failures that create risk for the application and the people who depend on it. Not the foundational benchmarks that rank base models on leaderboards.
12 methods
Compare the AI feature with the process it replaces — the search box, the rules engine, the form, the person — on the same work, so 'it works' is measured against what users had before rather than against nothing.
Ask the same question in each language the system serves and check the substance of the answer is the same — the facts, the refusal, the recommendation and the caveats, not just the topic.
Check that the system actually does what it was told — every constraint rather than most of them, still at turn forty, and with the right source winning when two instructions conflict.
Exercise the seams where the model meets the rest of the system — tool and function contracts, MCP handshakes, retrieval calls, API auth — to catch connectivity and contract failures end to end.
Before moving a working system to a newer model, replay what it did on the old one and check the new version keeps every behaviour you rely on — better on average is not the same as no worse anywhere.
Treat durable memory as a system with a lifecycle — what got written, whether it returns at the right moment, whether it was corrected when the world moved, and what one wrong entry costs three sessions later.
A fast, shallow pass on every build — a handful of canonical prompts and health checks — whose only job is to fail loudly on gross breakage before anything deeper runs.
Check whether the answer moves when the user signals what they want to hear — pushing back on a correct answer, stating a false belief up front, or showing which conclusion they prefer — while the facts stay exactly the same.
Deliberately probe whether a model abandons a correct answer or endorses a false one when the user pushes back, to catch approval-seeking behavior that plain accuracy evals miss.
Instrument the whole request — prompt, retrieval, model call, tool calls, tokens, latency, outcome — so real production behaviour becomes evidence you can query, sample, and score.
Push volume, length, and concurrency past what the system enforces — unbounded prompts, recursive agent loops, or scripted request floods — to find where cost or capacity, not correctness, breaks first.
Test the deterministic code around the model — prompt builders, output parsers, schema validators, tool wrappers — in isolation, with exact assertions, the way you'd test any software.
