Perturbation testing

How it works

A robust system should be indifferent to changes that don't change meaning: a stray typo, extra whitespace, a synonym, a reordered clause. Perturbation testing applies these small transformations at scale and flags every case where the answer moves. It is a focused, robustness-oriented cousin of metamorphic testing — the relation is simply 'meaning-preserving in, same answer out' — and it exposes the prompt-sensitivity that quietly undermines reproducibility.

When to use it

Robustness hardening; quantifying prompt-sensitivity; regression-guarding inputs that users will phrase many different ways — especially a support or search-style interface, where the same underlying question arrives in dozens of real phrasings and a brittle system answers some well and others badly for no principled reason.

Limitations

You must ensure the perturbation truly preserves meaning — an over-aggressive change (a typo that genuinely alters the question) creates false positives — and it detects instability, not which of the diverging answers, if either, is correct.

Cite this