Underspecification & premature-commitment testing

How it works

Most evaluation assumes a well-formed task and asks whether the answer was right. Real requests arrive underspecified — over half of the open-domain questions in one widely used benchmark turned out to be ambiguous — and the failure this method targets is not a wrong fact but a confident answer to a question nobody asked, or an action taken before the facts that determined it were in hand. The corpus is built from inputs whose correct handling is to establish something first, each labelled with what specifically had to be established, and every response is scored into one of four behaviours: asked a targeted clarifying question, proceeded while stating the assumption it chose, proceeded silently under one reading, or — for anything with side effects — acted before the fact was in hand. Matched pairs sharpen it: take a fully specified task and ablate the one detail that makes it decidable, and the behaviour is expected to change, which turns a vague complaint about overconfidence into a measurable relation. A control set of genuinely well-specified tasks runs alongside, because the failure has a mirror image — a system that interrogates every request is unusable, and clarification rate is otherwise trivial to optimize upward. For agents the same corpus covers acting before reading: editing a file it never opened, calling a tool whose preconditions it never checked, settling on a diagnosis from the first log line it saw.

When to use it

Any system taking open-ended requests from people who know their own domain better than they can specify it; agents with side effects, where committing early costs real work rather than one bad paragraph; after any prompt or model change that pushes toward decisiveness, since an instruction not to bother the user with questions moves this number without moving any accuracy metric; and next to refusal testing, because asking and refusing are both failures to deliver an answer and blur together in aggregate reporting.

Limitations

The per-item label — what genuinely had to be established before proceeding — is the method, and annotators disagree about it, so the rubric's quality is the measurement's ceiling. It is also gameable in the mirror direction, which is why the control set and the cost of the extra turn belong in the same report as the clarification rate. Context can be established without asking anyone — by reading the file, running the query, checking the schema — so a scoring scheme that counts only clarifying questions penalizes the system that went and found out for itself. And ambiguity is audience-relative: what is underspecified coming from a stranger may be perfectly clear coming from a returning user whose standing context the system already holds.

Cite this