Differential testing

How it works

When two comparable systems — two providers, two versions of the same model, or the same model at two temperature settings — disagree on the same input, at least one of them is wrong, or the behavior is genuinely unstable either way. Running an identical prompt set across both and diffing the outputs surfaces regressions after an upgrade, provider-specific quirks worth knowing before committing to one, and prompts where the 'right' answer turns out to be less settled than assumed — all without a labelled oracle for the whole set, since divergence itself is the signal that narrows down which cases are worth a closer look.

When to use it

Regression testing across model or provider upgrades before rolling one out; comparing candidate providers or model families during selection; flagging unstable or version-sensitive behavior on prompts a system depends on.

Limitations

Agreement between the two systems doesn't prove correctness — both can share the same training-data blind spot and be confidently wrong in the same way. It narrows down which cases need a closer look; it doesn't resolve them, so differential testing is best combined with an oracle (or a human) applied specifically to the divergent cases rather than trusted as a standalone verdict.

Cite this