Quick answer
Data poisoning is an attack that corrupts the data an AI system learns from or relies on, so the model behaves the way the attacker wants. It can target training data, fine-tuning data or the documents a retrieval-augmented system searches, and it can plant hidden backdoors that only activate on a trigger phrase.
On this page
How It Works
In training and fine-tuning, an attacker who can influence the dataset, for example through public web content, user feedback loops, open datasets or a compromised data pipeline, inserts examples that teach the model a wrong association or a backdoor. Research has shown that a small number of poisoned documents can be enough to implant a backdoor even in large models. In deployed systems, the more practical risk is poisoning the knowledge a model retrieves: a malicious or edited document in a shared drive, wiki or ticketing system that a RAG assistant later treats as trusted.
Why It Matters
Poisoned behaviour is hard to detect because the model works normally most of the time. The consequences range from biased or wrong answers in business decisions, to an assistant that leaks data or recommends a malicious link when a trigger appears, to a security classifier that has been trained to ignore a specific attack.
How to Test for It
- Map every data source that feeds training, fine-tuning, feedback and retrieval, and who can write to each.
- Test whether a low-privileged user can add or edit content that the AI system will later retrieve and trust.
- Plant benign canary documents with instructions and check whether the assistant follows or repeats them.
- Review the provenance and integrity controls on datasets and models from external sources.
- Evaluate model behaviour against a held-out test set after each training or fine-tuning run to spot unexpected shifts.
How to Reduce the Risk
Control who can write to training and retrieval sources, track data provenance and integrity, prefer curated sources over open feedback loops, scan and review content entering knowledge bases, separate trusted and untrusted documents in retrieval, evaluate models on fixed test sets after every update, and red team the system for both poisoning and prompt injection.
Regulatory and Framework Context
In Australia, ASD's guidance on engaging with AI and its joint publication on AI data security identify data poisoning as a key risk and recommend provenance tracking and integrity controls across the AI data supply chain.
Common questions
What is the difference between data poisoning and prompt injection?
Data poisoning corrupts what the model learns or retrieves ahead of time. Prompt injection manipulates the model through the input it receives at the moment of use. Poisoned retrieval documents often carry indirect prompt injections.
Can poisoning affect models we buy rather than train?
Yes, through the knowledge bases and feedback you connect to them, and through any fine-tuning you do. Retrieval poisoning is the most practical risk for most organisations.
Change log
- 13 November 2026First published.