Data poisoning is an attack on an AI model in which someone deliberately corrupts what the model learns from, so that it behaves the way the attacker wants. The aim might be to make it less accurate in general, to make it fail on particular inputs, or to plant a hidden "backdoor" that only activates when a specific trigger appears. The term covers tampering with training data and, more broadly, tampering with the model itself before it reaches you. For most organisations, the practical risk isn't someone poisoning a model you trained. It is using a model, fine-tune or dataset that someone else has already poisoned.
What is data poisoning?
A machine learning model is shaped by the data it is trained on and by any later fine-tuning. If an attacker can influence that data, they can influence the model. Unlike most attacks, poisoning happens before the system is deployed. Once the model is live, the bad behaviour is part of how it works, and it is invisible until the right input comes along.
The idea isn't new. Researchers have shown poisoning attacks on spam filters, image classifiers and malware detectors for years. What has changed is scale. Large language models are trained on enormous volumes of web data that nobody can fully review, many organisations download open-weight models and fine-tunes from public hubs, and those models are increasingly connected to tools that can take actions.
Types of data poisoning attack
Availability attacks
The attacker degrades the model in general, so that it becomes less accurate or less useful. These are the easiest to notice, because performance drops in testing.
Targeted attacks
The model works normally except on specific inputs the attacker cares about. For example, a classifier might mislabel one company's documents, or a content filter might let through one phrasing.
Backdoor (trigger) attacks
The model behaves normally until it sees a particular trigger, such as a word, phrase or pattern, and then does something the attacker chose. Because it passes ordinary tests, this is the most dangerous kind. A backdoored model that drives an AI agent could, when triggered, run a command, leak data or approve something it shouldn't.
Model tampering after training
Strictly speaking this isn't poisoning the data, but it has the same effect and is often discussed alongside it. Here the attacker edits the model's weights directly. In November 2026, Microsoft Threat Intelligence showed that changing fewer than ten neurons in a capable agent model can implant a trigger-activated backdoor that still "passes all evaluations". Microsoft says it hasn't seen attackers use the technique yet. Our blog on the research covers what it means for organisations deploying agents.
How does data poisoning happen in practice?
- Poisoned public data. Attackers plant content on websites, code repositories or forums they expect to be scraped into training sets.
- Compromised datasets. Public datasets are edited, or look-alike versions are published under similar names.
- Malicious fine-tunes and adapters. Someone uploads a fine-tuned version of a popular model with a backdoor added, and it gets downloaded because it looks useful.
- Insider or supply chain access. Someone with access to your training pipeline, labelling process or model registry changes data or weights.
- Feedback loops. Systems that keep learning from user input, such as feedback buttons or retraining on conversations, can be steered by people who feed them crafted input over time.
- Poisoned retrieval sources. In retrieval-augmented generation (RAG), the model is fed documents at query time. Planting instructions in those documents isn't training-data poisoning, but the effect is similar, and it is usually called indirect prompt injection.
Data poisoning vs prompt injection
The two are often confused. Data poisoning changes the model itself, through its training data or weights, so the bad behaviour is permanent until the model is replaced. Prompt injection leaves the model untouched and tries to manipulate it at the time of use, through the text it is given. Poisoning is harder to pull off and harder to detect. Prompt injection is easier but has to be repeated each time. Both are reduced by the same runtime control: limiting what the model is allowed to do.
Why it matters for security teams
A poisoned model is a supply chain problem. If you download a model and connect it to an agent with access to email, files, a shell or a cloud subscription, a hidden trigger becomes a way to run actions inside your estate with the agent's privileges. Unlike a malicious software package, you can't read the model to check it. And because the backdoor is designed to pass evaluations, a model that tested well tells you less than you might think.
How to check the models you deploy
You can't prove a model is clean, but you can make poisoning much harder to get through and much less harmful if it does. Microsoft's research sets out a detailed list. This checklist follows it.
- Keep an inventory. Record every model in use, where it came from, the base model it claims to be derived from, and its file hashes.
- Use trusted sources. Download from the original publisher, verify published hashes, and treat third-party fine-tunes, merges, quantisations and adapters as untrusted until checked.
- Use a private registry. Store approved models in an access-controlled registry and stop applications pulling models from public hubs at runtime.
- Compare against the base model. Legitimate fine-tunes change weights broadly and slightly. Implanted backdoors tend to change a few places a lot. Diffing a checkpoint against its declared base can expose that, and models without a verifiable base should be rejected.
- Scan files for code-level risks too. Some model file formats can carry executable code. Model scanning tools, such as Microsoft Defender for Cloud's AI model security, look for those risks, but Microsoft is clear they don't replace a weight-level check.
- Test for backdoors. Add trigger-hunting and backdoor scenarios to AI red teaming, alongside normal evaluation.
- Protect your own training data. If you fine-tune, control who can change datasets and labels, keep versions, and record where each training example came from.
- Limit the blast radius. Give agents least-privilege access, run generated code in sandboxes, restrict outbound network access, and require human approval for high-impact actions. This is the control that still works when every check above has been beaten.
- Monitor behaviour. Log the agent's tool calls, network connections and access requests, and alert on anything outside its normal pattern.
Frequently asked questions
Is data poisoning the same as model poisoning?
The terms overlap. "Data poisoning" usually means corrupting training or fine-tuning data. "Model poisoning" is often used for tampering with the model or its updates directly. The defences are largely the same.
Can you remove poisoning from a model?
Sometimes, through retraining or targeted fine-tuning, but there is no reliable way to confirm it is gone. The safer approach is to replace the model with one from a verified source.
Does this affect AI services I use through an API?
Less directly. With a hosted model from a major provider, protecting the training pipeline is the provider's job. You still control what data you feed it, what it is connected to and what it is allowed to do, and those controls matter just as much.
Has data poisoning been used in real attacks?
Poisoning has been shown many times in research, and there have been reports of malicious models uploaded to public hubs. Microsoft says it has not seen attackers use the specific weight-editing technique it published in November 2026. The time to put controls in place is before that changes.
Who should own this in an organisation?
Usually nobody does, which is the problem. Model sourcing tends to sit with data science, while the risk sits with security. Someone needs to own the approval process for models in the same way someone owns approval for software.
How Wizard Cyber helps
We use AI throughout our own security operations, so model supply chain is a question we answer for ourselves. Our vCISO service can help you set the policy for which models are approved and what they may touch, and our SOCs, staffed 24/7, can monitor AI agent identities and behaviour alongside the rest of your Microsoft estate. To see how we approach AI-driven security operations, book a demo.