Data poisoning is a cyberattack where malicious actors intentionally manipulate, corrupt, or inject false information into the training dataset of an artificial intelligence (AI) or machine learning (ML) model. Unlike traditional hacks that exploit software code after launch, data poisoning attacks the foundational learning stage itself because the bad data is baked into training, the model learns incorrect patterns, leading to flawed decisions, systemic biases, or hidden security vulnerabilities once it’s live.
How Data Poisoning Works
- Injection. Attackers add fake or mislabeled entries into public or scraped datasets used for training.
- Modification. Attackers alter existing data points or swap correct labels for incorrect ones, a technique often called label flipping.
- Deletion. Attackers remove critical records to create blind spots in the model's understanding of certain cases or categories.
Common Types of Data Poisoning Attacks
Backdoor attacks.
Hackers embed a hidden trigger or pattern in the training data. The model behaves normally until it encounters that specific trigger, at which point it’s forced to fail or act maliciously – for instance, a specific image watermark that causes a facial recognition system to misclassify.
Targeted attacks.
The poisoning is engineered to force one specific, precise error, such as making a facial recognition system consistently mistake one particular person for someone else.
Non-targeted attacks.
The goal is general disruption – injecting enough bad data to degrade the overall accuracy and reliability of the system over time, without aiming at any single outcome.
Why Data Poisoning Is So Dangerous
Stealthy.
Poisoned entries usually look clean, valid, and properly formatted, which makes them extremely difficult to catch during standard software testing or manual review.
Persistent.
Once a model has learned corrupted data, standard runtime security filters and firewalls can’t remove the resulting behavior, because the flaw is built into the neural network’s learned weights, not into a piece of code sitting on top of it.
An expanding attack surface.
The rise of generative AI, retrieval-augmented generation (RAG), and fine-tuning on scraped web data or vector databases gives attackers more entry points than ever – any data source a model learns from, or retrieves from at inference time, is a potential poisoning vector.
Data Poisoning vs. Adversarial Attacks
The two are closely related and often confused. Data poisoning happens before or during training – it corrupts what the model learns. An adversarial attack happens after training, at inference time – it’s a carefully crafted input designed to fool an already-trained model into a wrong output, without ever touching the training data itself. A poisoned model can be fooled by completely normal inputs, because the flaw lives in what it learned; an adversarially attacked model is otherwise sound but gets tricked by one specific, deliberately engineered input.
Detecting and Preventing Data Poisoning
- Data provenance and lineage tracking, so every training dataset's source can be audited and unusual or unverified additions can be flagged before they're used.
- Anomaly detection on training data itself, looking for statistical outliers or label inconsistencies before training runs, rather than only monitoring the model's behavior after deployment.
- Robust training techniques, such as methods designed to reduce a model's sensitivity to a small number of poisoned samples relative to a much larger clean dataset.
- Strict access controls on training pipelines and vector databases, particularly for RAG systems, since an attacker who can write to a retrieval source can poison what the model "knows" without ever touching the original training run.
For AI systems that rely on retrieval-augmented generation (RAG), this also means treating the retrieval layer as part of the security boundary. The model may never be directly modified, yet compromised or manipulated content returned by a retrieval system can influence what the model uses to generate its response. As a result, securing the endpoints and data sources that supply model context is becoming an important part of protecting AI applications.
For AI applications that rely on APIs and external data sources, an AI security platform can add a runtime layer of protection around the application, helping secure the traffic and interactions that connect users, APIs, and AI systems.
Securing the Data Behind the Model
Data poisoning is uniquely difficult to deal with because it doesn’t behave like a conventional security flaw – there’s no patch to apply once a model has learned from corrupted data, because the vulnerability is baked into the model’s weights rather than sitting in a piece of code you can rewrite.
That’s exactly why prevention has to happen upstream, at the data layer: knowing where every training example and every piece of retrieved context actually came from, auditing for anomalies before training rather than only after deployment, and treating any pipeline that feeds a model – training set, fine-tuning data, or a RAG retrieval source – as security-sensitive infrastructure in its own right.
As models increasingly pull from external, less-controlled sources like scraped web data and live retrieval systems, the discipline of tracking and verifying what a model actually learns from is becoming as important as securing the model’s runtime API.
Frequently Asked Questions (FAQ)
1. What is LLM poisoning?
LLM poisoning refers to data poisoning specifically targeted at large language models – corrupting the text, documents, or feedback data used to pretrain or fine-tune an LLM so it learns biased, incorrect, or backdoored behavior. It’s the same underlying technique as data poisoning generally, applied to the specific training pipeline of a language model.
2. What is AI model poisoning?
“AI model poisoning” and “data poisoning” are generally used to mean the same thing: corrupting the data a model learns from so its resulting behavior is compromised. Some sources use “model poisoning” more narrowly to describe attacks on the model’s parameters or update process directly (relevant in federated learning, where multiple parties contribute model updates), as opposed to poisoning a centralized training dataset – worth clarifying which scenario applies to your architecture.
3. What is an example of data poisoning?
A commonly cited example is a spam filter trained on user-submitted “not spam” reports. An attacker who can submit many fake “not spam” reports mislabeling actual spam messages gradually teaches the filter to let similar spam through – the model’s behavior degrades because of what it learned, not because of a bug in the filter’s code.
4. What is AI context poisoning?
Context poisoning is a related but distinct concept, more relevant to retrieval-augmented generation (RAG) and AI agents: instead of corrupting the training data, an attacker manipulates the external content (documents, search results, retrieved context) a model reads at inference time, so its response is based on poisoned information without the model’s underlying training ever being touched.
5. What's the difference between data poisoning and an adversarial attack?
Data poisoning corrupts the training data before or during training, so the resulting model has learned something wrong. An adversarial attack targets an already-trained, otherwise-sound model with a specifically crafted input at inference time. Poisoning changes what the model knows; an adversarial attack exploits how the model responds to one deliberately engineered input.
Protect AI and LLMs, everywhere.
Discover AI & LLM threats, block prompt injection and jailbreak attacks, and enforce security policies at scale.
