Qlarify Labs
Catalog
How to find the limits of AI systems. Each entry is a repeatable testing technique — the durable knowledge, independent of any one model or version.
Qlarify Labs
How to find the limits of AI systems. Each entry is a repeatable testing technique — the durable knowledge, independent of any one model or version.
34 methods
Serve two variants — prompts, models, or settings — to comparable slices of real traffic and let live outcomes decide which behaves better.
Deliberately craft inputs designed to elicit failure — confusion, unsafe output, or broken constraints — to map the model's weak boundaries.
Judge the whole path an agent took, not just its final message — did it reach the goal, in how many steps, with which tools, and did it loop, stall, or wander.
Score the model against a fixed, known-answer dataset so performance becomes a number you can track across versions and compare across models.
Measure outcome disparities across a population of demographically varied inputs, reporting fairness as statistics rather than anecdotes.
Push inputs to limits — very long contexts, token boundaries, empty/extreme values — where behavior tends to degrade sharply.
Route a small slice of real traffic to a new model or prompt first, watch it closely, and widen or roll back based on what the canary shows.
Test whether a model's stated reasoning actually determines its answer, or is a post-hoc rationalization.
Deliberately inject failures — tool timeouts, malformed tool responses, truncated context, adversarial inputs — to test whether the system degrades gracefully and recovers.
Hold a prompt fixed while swapping a protected attribute (name, gender, ethnicity) — the output should not change. When it does, you've measured bias.
Run the same input across models or versions and treat divergence as a signal worth investigating.
Probe whether a deployed model can be cheaply queried to reconstruct its behaviour, training data, or a usable distilled copy — a confidentiality and IP attack surface.
Sample the model many times and test the distribution of its outputs — not any single answer — for drift, miscalibration, or instability.
Re-run a fixed suite against each release and over time, watching for the quiet regressions and capability decay that a one-off evaluation can't see.
Check generated claims against a trusted ground-truth source to catch hallucinations and fabricated citations.
Feed anomalous tokens, rare unicode, homoglyphs and malformed encodings to trigger out-of-distribution behavior.
Decompose an answer into its claims and check each one against the retrieved context — is the answer actually supported by what was retrieved, or did the model fill in the gaps?
Treat the safety filters around the model as their own system under test — measuring bypass rate on unsafe inputs, false-positive cost on benign ones, and what happens when the guardrail itself fails.
Deliberately steer the model toward fabrication — asking about non-existent entities or beyond its knowledge — to map where it invents instead of declining.
Exercise the seams where the model meets the rest of the system — tool and function contracts, MCP handshakes, retrieval calls, API auth — to catch connectivity and contract failures end to end.
Check that the model's outputs obey the rules of logic — valid inference, transitivity, symmetry, no self-contradiction — across related questions.
Test without an oracle by checking relations between the outputs of related inputs, instead of judging any single output in isolation.
Use a strong model as an approximate oracle — grading, comparing, or fact-checking another model's output where no cheap ground-truth label exists.
Plant a specific fact at varying depths in a long context and test whether the model can retrieve it from each position.
Apply small, meaning-preserving changes to an input — typos, spacing, paraphrase, reordering — and check that the output stays stable. When it doesn't, you've measured brittleness.
Embed adversarial instructions in user input or retrieved/tool content to test whether the model follows attacker text over its system policy.
Specify invariants the output must always satisfy, then generate many inputs automatically and check the invariant on each.
Measure the retriever on its own terms — did the right documents come back, and how far up the ranking — before judging anything the model wrote with them.
Ask the same question multiple times (or multiple ways) and measure how often the answers agree.
A fast, shallow pass on every build — a handful of canonical prompts and health checks — whose only job is to fail loudly on gross breakage before anything deeper runs.
Deliberately probe whether a model abandons a correct answer or endorses a false one when the user pushes back, to catch approval-seeking behavior that plain accuracy evals miss.
Walk inputs across a decision boundary — refusal, classification, confidence cutoff — to find exactly where the model's behaviour flips, and whether it flips in the right place.
Instrument the whole request — prompt, retrieval, model call, tool calls, tokens, latency, outcome — so real production behaviour becomes evidence you can query, sample, and score.
Test the deterministic code around the model — prompt builders, output parsers, schema validators, tool wrappers — in isolation, with exact assertions, the way you'd test any software.