Sensitive information disclosure testing

How it works

LLMs can memorize and later reproduce fragments of what they were trained, fine-tuned, or grounded on, and a long-lived conversation or agent session accumulates context an attacker can try to pull back out. Testing drives both angles directly: extraction prompts targeting known training-set content (names, addresses, credentials seen during training), and session-level probing that asks the model to summarize, repeat, or translate anything said earlier, hoping something sensitive slips through. Seeded canary strings — planted, traceable values that shouldn't appear in any output — turn a vague worry about leakage into a deterministic pass/fail. Output scanning for PII/secret patterns runs as the oracle in both cases.

When to use it

Any system trained, fine-tuned, or grounded on data that might contain regulated or proprietary information; any agent or chat session that accumulates PII, credentials, or business data in context.

Limitations

Only catches what the probe corpus and the output-scanning patterns cover — novel PII formats and unanticipated extraction phrasings slip through. A clean run proves nothing was extracted by the tested attacks, not that nothing is memorized.

Cite this