Training-data & model poisoning detection

How it works

Poisoning attacks corrupt a model indirectly, through the data it learns from or retrieves against, rather than through its inputs at inference time. Detection works two ends of the pipeline: on the corpus, statistical outlier and duplication analysis flags examples that look engineered rather than sampled, and known trigger-pattern scanning looks for a rare token or phrase suspiciously correlated with an anomalous label. On the model, holdout evaluation against candidate canary triggers after every fine-tune, and behavioural diffing before/after ingesting a new bulk data source, catch a backdoor that corpus inspection alone missed. The RAG variant is the same idea applied to a retrieval index: a poisoned document injected into the corpus that a normal query will surface and the model will trust.

When to use it

Before shipping a model fine-tuned or RAG-augmented on data you don't fully control — scraped, crowdsourced, or user-submitted; after ingesting any new bulk data source.

Limitations

Detects known trigger patterns and statistical anomalies well; a subtle, low-frequency backdoor engineered to evade exactly this kind of scan can still get through. Canary-trigger holdouts only test the triggers you thought to include.

Cite this