Application evals — testing AI systems where the model is one component of a larger piece of software. Each entry is a repeatable technique for catching regressions, measuring accuracy, and finding the failures that create risk for the application and the people who depend on it. Not the foundational benchmarks that rank base models on leaderboards.
20 methods
Compare the AI feature with the process it replaces — the search box, the rules engine, the form, the person — on the same work, so 'it works' is measured against what users had before rather than against nothing.
Check that the system speaks the way the organization wants to be seen — in voice, in role, and above all in what it promises — and never commits the organization to a policy, price or exception that doesn't exist.
Route a small slice of real traffic to a new model or prompt first, watch it closely, and widen or roll back based on what the canary shows.
Test whether the reasons a system gives — its step-by-step reasoning, or the explanation it shows a user for a decision — actually determine its answer, or are a plausible story told afterwards.
Compare what an agent says it did with what the trace shows it did — 'tests pass', 'file saved', 'email sent' — and treat every mismatch as a failure, even when the task itself went fine.
Hold a prompt fixed while swapping a protected attribute (name, gender, ethnicity) — the output should not change. When it does, you've measured bias.
Ask the same question in each language the system serves and check the substance of the answer is the same — the facts, the refusal, the recommendation and the caveats, not just the topic.
Re-measure the accuracy, robustness and per-group performance a provider declares for its system — under the conditions the declaration names — and check the system actually meets what its documentation says.
Run the same input across models or versions and treat divergence as a signal worth investigating.
Re-run a fixed suite against each release and over time, watching for the quiet regressions and capability decay that a one-off evaluation can't see.
Check that the system actually does what it was told — every constraint rather than most of them, still at turn forty, and with the right source winning when two instructions conflict.
Check that the model's outputs obey the rules of logic — valid inference, transitivity, symmetry, no self-contradiction — across related questions.
Test by transforming an input in a way whose effect on the output is known in advance, then checking the relation holds — no ground-truth label required.
Before moving a working system to a newer model, replay what it did on the old one and check the new version keeps every behaviour you rely on — better on average is not the same as no worse anywhere.
Treat durable memory as a system with a lifecycle — what got written, whether it returns at the right moment, whether it was corrected when the world moved, and what one wrong entry costs three sessions later.
Apply small, meaning-preserving changes to an input — typos, spacing, paraphrase, reordering — and check that the output stays stable. When it doesn't, you've measured brittleness.
Specify invariants the output must always satisfy, then generate many inputs automatically and check the invariant on each.
Ask the same question multiple times (or multiple ways) and measure how often the answers agree.
Measure whether a model that grades, routes to, or recommends other models favours its own family — by re-running the same comparison with authorship hidden, swapped, and deliberately mislabelled.
Check whether the answer moves when the user signals what they want to hear — pushing back on a correct answer, stating a false belief up front, or showing which conclusion they prefer — while the facts stay exactly the same.
