Differential testing

How it works

When two comparable systems disagree on the same input, at least one is wrong — or the behavior is unstable. Comparing across providers, or across versions of one model, surfaces regressions and version-specific quirks cheaply, again without a labelled oracle.

When to use it

Regression testing across model upgrades; provider selection; flagging unstable behaviors.

Limitations

Agreement doesn't prove correctness (both can be wrong the same way). Best combined with an oracle on the divergent cases.

Cite this