Metamorphic testing

How it works

Instead of asking 'is this single answer correct?', metamorphic testing defines a relation between an input transformation and its expected effect on the output, then checks that the relation holds across many transformed inputs. The vocabulary maps directly onto LLM behavior: invariance (paraphrasing a factual question should not change the answer), monotonicity (adding supporting evidence should not lower a stated confidence score), and structural relations (translating a math word problem into another language should not change the numeric answer). A concrete pass might take one seed prompt, generate a dozen paraphrases and reorderings, and flag every case where the invariance relation breaks. Because the check is relational rather than absolute, it needs no labelled ground truth — which is exactly what makes it powerful for LLMs, where ground truth is expensive or doesn't exist for open-ended tasks.

When to use it

When you have no cheap oracle but you do know how outputs *should* relate across transformed inputs — paraphrase, reorder, add irrelevant context, swap a protected attribute, translate and back. Especially useful as a regression check after a model or prompt upgrade, where the absolute correctness of an answer may be unknown but a break in an established relation is itself the signal.

Limitations

The method is only as good as the relations you design; a relation that doesn't actually hold in the domain (assuming strict invariance where some legitimate variation is normal) produces false positives. It detects that a relation broke, not which of the two outputs — if either — is correct, so a violation still needs a follow-up pass to interpret. Relations also have to be re-validated after a model upgrade, since what counts as acceptable variance can shift.

Cite this