Scalable Extraction of Training Data from (Production) Language Models

Our summary

Shows that prompting aligned ChatGPT to endlessly repeat a token makes it diverge from chat-style output and emit memorized training data — including PII — at ~150x the normal rate, recovering over ten thousand unique training examples for about $200.

Why it matters

Both a memorization/privacy failure and a model-extraction surface: a deployed, aligned model leaks its own training corpus under the right probe.

Cited by these methods

Related findings (1)

• Verbatim training data extracted from a deployed chatbotHigh

  A 'divergence' attack made aligned ChatGPT abandon its chat format and emit memorized training data verbatim, recovering thousands of examples for about $200.

Cite this