Restatement fidelity testing (paraphrase drift)

How it works

Metamorphic testing paraphrases the input and asks whether the answer holds. This is the opposite direction: the system produces the paraphrase, and the question is whether the source's meaning survived it. Summaries, meeting notes, ticket triage, a memory written about a conversation, one agent restating a task for another — all of them re-express something a person or a prior step actually said, and all of them can drift while reading as faithful. The drift has recognizable shapes, and each is scored separately rather than as one faithfulness score: a claim added or dropped; hedging hardened or softened ('might' becoming 'will', 'one user reported' becoming 'users report'); attribution moved, so a speaker's words become the system's own assertion or the reverse; scope and quantifiers widened; and — the one that reads most convincingly and is hardest to catch — a motive or reason supplied for a decision that the source never gave. Build the corpus from real sources paired with a per-item label of what was load-bearing in them, then score restatements against it, by entailment model, judge model, or human. The relay variant measures accumulation: pass the same content through several hops, agent to agent, and plot how far it has moved by the third restatement — drift per hop is a number a single-pass check never shows.

When to use it

Summarization, recaps, meeting notes and any feature that quotes a person back to themselves or to a third party; multi-agent pipelines where one model restates a task, a constraint, or a user's request for another and the original text is never seen again; before a system is allowed to store its own summary of a conversation as durable state, since a distorted memory is re-read as fact long after the source is gone; and anywhere a paraphrase carries the weight of a decision — a triaged ticket, a routed request, an escalation note.

Limitations

Fidelity is not binary, because compression requires dropping something: the rubric has to separate permitted compression from meaning change, and that judgement is exactly where annotators disagree, so the label quality is the measurement's ceiling. A judge or entailment model used as the checker inherits the same paraphrasing tendencies it is being asked to detect, which makes calibration against human labels a prerequisite rather than a refinement. And the hardest case is invisible from the text alone — a restatement can be faithful to every word and still miss what the speaker meant, which no source-comparison method can recover.

Cite this