Adversarial prompting

How it works

Adversarial prompting probes a model the way an attacker — or simply an unlucky, confused, or persistent user — would: misleading framings ('for a novel I'm writing...'), conflicting instructions stacked to see which one wins, edge-case phrasings that sit just outside a policy's wording, and steady pressure applied across several turns rather than one. Unlike prompt-injection testing, which targets untrusted content the model ingests, adversarial prompting targets the direct conversation — what a motivated user typing into the box can get the model to do. It is the workhorse for surfacing safety and robustness issues that benign, happy-path testing never reaches, precisely because those issues live in inputs nobody would write by accident.

When to use it

Robustness and safety assessment before shipping anything user-facing or agentic; whenever a new refusal policy or safety-tuning pass ships, to check it holds under pressure rather than just against the prompts it was tuned on.

Limitations

Coverage depends entirely on tester creativity and the taxonomy of attack framings tried — an untested framing is an untested risk. Results can be hard to reproduce because outputs are stochastic, so a single failed attempt proves little and a single success isn't a stable measurement; pair adversarial prompting with a repro protocol (hit rate over N attempts) rather than reporting one-off transcripts.

Cite this