Application evals — testing AI systems where the model is one component of a larger piece of software. Each entry is a repeatable technique for catching regressions, measuring accuracy, and finding the failures that create risk for the application and the people who depend on it. Not the foundational benchmarks that rank base models on leaderboards.
13 methods
Check that the system meets the transparency duties the law places on its provider and deployer — people told they are talking to an AI, generated content marked as such — in every channel, at the latest at the first interaction, and every time rather than most of the time.
Deliberately craft inputs designed to elicit failure — confusion, unsafe output, or broken constraints — to map the model's weak boundaries.
Route a small slice of real traffic to a new model or prompt first, watch it closely, and widen or roll back based on what the canary shows.
Probe whether a deployed model can be cheaply queried to reconstruct its behaviour, training data, or a usable distilled copy — a confidentiality and IP attack surface.
Feed anomalous tokens, rare unicode, homoglyphs and malformed encodings to trigger out-of-distribution behavior.
Treat the safety filters around the model as their own system under test — measuring bypass rate on unsafe inputs, false-positive cost on benign ones, and what happens when the guardrail itself fails.
Check whether a persuasive or engagement-driven system uses deceptive or manipulative techniques to steer people — and whether it treats users made vulnerable by age, disability or a social or economic situation differently in ways that exploit that vulnerability.
Treat the base model, fine-tuning data, embeddings, plugins, and every third-party package pulled into the pipeline as a software supply chain — provenance-checked and version-pinned, not merely capability-tested.
Treat the model's output as untrusted input to whatever consumes it next — feed it into the real downstream sink and check the sink's own defences, not the model's judgement.
Embed adversarial instructions in user input or retrieved/tool content to test whether the model follows attacker text over its system policy.
Probe the model and its outputs for PII, credentials, or proprietary data it was never meant to reveal — memorized training data, leaked context, or a secret pasted earlier in the same conversation.
Try to get the model to reveal its own system prompt or hidden instructions, then check whether that prompt held anything — a credential, an access-control rule, unfiled business logic — that mattered only because nobody expected it to leak.
Probe the RAG pipeline's storage layer, not just its retrieval quality — can embeddings be inverted back to source text, can one tenant's query surface another tenant's documents, can a crafted document poison what gets embedded and returned.
