Perturbation testing

How it works

A robust system should be indifferent to changes that don't change meaning: a stray typo, extra whitespace, a synonym, a reordered clause. Perturbation testing applies these small transformations at scale and flags every case where the answer moves. It is a focused, robustness-oriented cousin of metamorphic testing — the relation is simply 'meaning-preserving in, same answer out' — and it exposes the prompt-sensitivity that quietly undermines reproducibility.

When to use it

Robustness hardening; quantifying prompt-sensitivity; regression-guarding inputs that users will phrase many different ways.

Limitations

You must ensure the perturbation truly preserves meaning — an over-aggressive change creates false positives — and it detects instability, not which of the diverging answers is correct.

Cite this