Guardrail & moderation-layer testing

How it works

Most deployed AI systems wrap the model in a separate policy layer: input classifiers, output moderation, PII and secret redaction, topic and jailbreak filters. That layer is software with its own failure modes, and testing the model alone never exercises it. Guardrail testing drives it directly with two labelled corpora — genuinely violating inputs, and benign inputs that merely look violating — and reports both error rates, because a filter tuned only for bypass rate blocks the innocent. It also probes the layer's own engineering: does it fail open or closed when the classifier times out, is it applied to streamed output as well as buffered, can it be sidestepped by encoding or a tool call that never routes through it.

When to use it

Any system with moderation, refusal policy, or content filters in front of or behind the model; after a policy change; when tuning filter aggressiveness against a real complaint rate.

Limitations

Only measures the categories your labelled corpora cover, and the adversarial half ages quickly as new bypass techniques appear. It scores the guardrail, not the model — a well-guarded system can still be built on a model that shouldn't be trusted with the task.

Cite this