Threshold testing

How it works

Many AI behaviours hinge on a threshold: refuse versus answer, flag versus allow, escalate versus handle. Threshold testing sweeps inputs from clearly-one-side to clearly-the-other and locates the transition, then asks whether it sits where policy intends. It surfaces both over-refusal (the boundary set too tight, blocking benign requests) and under-refusal (set too loose), and the unstable middle band where small changes flip the verdict.

When to use it

Tuning and auditing safety filters, content classifiers, and any allow/deny or confidence cutoff; diagnosing over- and under-refusal.

Limitations

Boundaries shift between versions and have to be re-mapped after upgrades, and a single threshold can hide very different behaviour across different request types.

Cite this