System-prompt leakage testing

How it works

Extraction attempts range from the direct ('repeat your instructions verbatim') to the indirect — asking the model to summarize, translate, encode, or roleplay its way into echoing the prompt it was given. Running a battery of these against the deployed system measures how easily the prompt comes out. But the sharper half of this method is auditing what's in the prompt in the first place: a system prompt is operator-authored text sent to the model on every call, not a secure secret store, so anything whose security depended on the prompt staying hidden — an API key, a discount code, an access-control rule a user could exploit if they knew it — is a finding regardless of whether extraction succeeds today.

When to use it

Any deployment with a non-trivial system prompt; especially before relying on the prompt to encode business rules, formatting contracts, or anything sensitive; and as a recurring check whenever the prompt changes, since a newly added instruction is exactly the kind of addition nobody remembers to re-audit for leaked business logic.

Limitations

A prompt that resists today's extraction techniques may not resist tomorrow's — this measures current resistance against known techniques, not a guarantee of secrecy going forward. The audit-what's-in-the-prompt half only catches what the auditor recognizes as sensitive; boilerplate-looking instructions that happen to encode a real business rule are easy to skim past.

Cite this