Qlarify Labs
Catalog
How to find the limits of AI systems. Each entry is a repeatable testing technique — the durable knowledge, independent of any one model or version.
Qlarify Labs
How to find the limits of AI systems. Each entry is a repeatable testing technique — the durable knowledge, independent of any one model or version.
6 methods
Test whether a model's stated reasoning actually determines its answer, or is a post-hoc rationalization.
Check that the model's outputs obey the rules of logic — valid inference, transitivity, symmetry, no self-contradiction — across related questions.
Test without an oracle by checking relations between the outputs of related inputs, instead of judging any single output in isolation.
Apply small, meaning-preserving changes to an input — typos, spacing, paraphrase, reordering — and check that the output stays stable. When it doesn't, you've measured brittleness.
Ask the same question multiple times (or multiple ways) and measure how often the answers agree.
Deliberately probe whether a model abandons a correct answer or endorses a false one when the user pushes back, to catch approval-seeking behavior that plain accuracy evals miss.