Qlarify Labs
Evals
How we evaluate AI systems to find their limits. Each entry is a repeatable testing technique — the durable knowledge, independent of any one model or version.
Qlarify Labs
How we evaluate AI systems to find their limits. Each entry is a repeatable testing technique — the durable knowledge, independent of any one model or version.
6 methods
Test whether a model's stated reasoning actually determines its answer, or is a post-hoc rationalization.
Check that the model's outputs obey the rules of logic — valid inference, transitivity, symmetry, no self-contradiction — across related questions.
Test without an oracle by checking relations between the outputs of related inputs, instead of judging any single output in isolation.
Apply small, meaning-preserving changes to an input — typos, spacing, paraphrase, reordering — and check that the output stays stable. When it doesn't, you've measured brittleness.
Ask the same question multiple times (or multiple ways) and measure how often the answers agree.
Deliberately probe whether a model abandons a correct answer or endorses a false one when the user pushes back, to catch approval-seeking behavior that plain accuracy evals miss.