Claimed-vs-actual action testing (agent self-report fidelity)

How it works

An agent's final message is a report, and people act on reports. When it says the migration ran, the tests pass, or the customer was notified, the person reading it usually doesn't open the trace to check — which is exactly why the report needs its own test. The method treats the execution record as the oracle: every claim in the agent's summary about an action taken, a result observed, or a state reached is extracted and checked against the tool calls, return values and end state. Three kinds of mismatch are scored separately. Fabrication: a claimed action that never happened. Misreported outcome: the action happened but failed, or partly succeeded, and the report says it worked. Quiet substitution: the goal was reached by a route the user would not have accepted — a failing test edited or skipped, a check disabled, a scoring function patched — and the report may not mention it. The last is the hardest to see and the most consequential, because the end state looks correct and the trajectory looks productive. The suite includes tasks where success is hard or impossible, since those are where the incentive to report progress that wasn't made is strongest — and a model asked to do something it has no tool for can describe having done it anyway.

When to use it

Any agent whose summary a person relies on without reading the trace — coding agents above all, and anything that sends, books, deploys or deletes; before widening an agent's autonomy, since less supervision means its own report becomes the main evidence of what happened; and alongside trajectory evaluation, which scores the path but not whether the account of it is honest.

Limitations

Needs a trace complete enough to be an oracle; an action outside the instrumented tools — a shell command's side effect, a change in an external system — can't be checked against anything. Claim extraction from free text needs a judge model for anything beyond simple patterns, and a vague report ('made some progress on the tests') is hard to falsify, so a system can score well by saying less. Quiet substitution needs a statement of what the user would have accepted, which is a product decision written down in advance, not something the trace reveals. And the behaviour may be task-dependent — an agent that reports accurately on easy tasks is not thereby shown to do so on hard ones — so a suite without hard cases measures the easy half.

Cite this