Data Poisoning

What data poisoning is

When a model trains on data, it has no way to verify whether that data is trustworthy. It treats every sample equally — a mislabeled image counts the same as a correctly labeled one. Data poisoning exploits this blind spot. An attacker who can influence what goes into a training dataset can shape what the model learns, without ever touching the model’s weights directly.

Think of it like adulterating a recipe test. If a chef uses one bad ingredient during every tasting session, their calibrated palate slowly adjusts to consider that off-note normal. The model does the same thing: it learns the poisoned distribution as ground truth.

The attack takes two broad forms. Availability attacks aim to degrade overall model performance — flooding the training set with noise or wrong labels until the model becomes unreliable. Backdoor attacks (also called Trojan attacks) are more surgical: the attacker injects a small number of samples that teach the model to behave correctly in most cases, but produce a specific, attacker-chosen output whenever a hidden trigger pattern appears in the input. Because the model performs normally on the test set, the backdoor survives validation undetected.

What sets data poisoning apart is that the damage is baked in before deployment. Unlike adversarial examples that fool a model at inference time, poisoning attacks compromise the model during training. By the time the model reaches production, the flaw is structural. Rolling back requires identifying the bad data, cleaning it, and retraining, which is expensive and sometimes impossible if data provenance was not tracked from the start.

Three factors determine how severe a poisoning attack can be: how much of the training data the attacker controls, whether the target labels or features are accessible, and whether the training pipeline includes any data validation. Even a small injection rate — as low as a fraction of a percent of the dataset — can be enough to embed a reliable backdoor.

How data poisoning is used in practice

The most common scenario product managers and developers encounter is not a targeted corporate attack — it is gradual, unintentional poisoning through poor data curation. When teams scrape the web to build training datasets without filtering, they often pull in mislabeled content, duplicate entries, or low-quality samples that skew the model’s learned behavior. A content moderation model trained on community-flagged data will underperform on any community that flags differently from the training demographic.

Intentional poisoning shows up in two production contexts. First, in models trained on user-contributed data — federated learning, fine-tuning pipelines that accept uploaded documents, or RAG systems where external knowledge bases can be altered. Second, in supply chain attacks where a third-party dataset or pre-trained model checkpoint was manipulated before distribution. A team that downloads a “pretrained image classifier” from a public repository has no guarantee the weights or the original training data were untampered.

Before fine-tuning on any external dataset, check its provenance — where it was sourced, who curated it, and whether checksums or version hashes are published. A dataset version-controlled with content hashes (see data-versioning) is far harder to poison silently than one downloaded from an anonymous URL.

The data poisoning reading path: threat model first, defense last

Start with the threat model itself: what corrupted training data does to model behavior explains why a poisoned model can pass every accuracy check and still carry someone else’s intent inside it. From there, backdoor triggers, clean-label, and label-flipping shows exactly why these attack types slip past manual review and standard data-cleaning pipelines — the mechanism any defense has to account for. RAG poisoning and agent-memory corruption then extends the same logic to systems that never retrain at all: a corrupted retrieval corpus or agent memory store changes behavior without touching a single model weight.

Once the threat model is clear, the ART, data provenance, and ML-BOM defense guide turns it into a build spec — provenance tracking, detector layering, and the compliance paperwork EU AI Act Article 11 starts requiring in August 2026. For the state of the actual threat, the 2026 shift from Nightshade to constant-sample attacks documents how the volume assumption collapsed — and closes with the ethics of deliberately poisoning training data as self-defense, worth reading before you decide whether a tool like Nightshade is a shield or an attack inside someone else’s threat model.

How data poisoning differs from red teaming, safety scoring, and data provenance

Three practices get pulled into the same conversation as poisoning, and each answers a different question.

Red teaming probes a model at inference time, hunting for prompts that make it misbehave right now; poisoning is planted at training time and can lie dormant through months of clean red-team runs before its trigger ever appears in a test set. A red team that never happens to type the trigger phrase will certify a backdoored model as safe.

Toxicity and safety evaluation scores what a model outputs, continuously, against a harm taxonomy — but a poisoned model’s normal outputs are exactly that: normal. Guard models grade behavior, not the data that produced it, so they have nothing to flag until the attack condition fires.

Data versioning is often assumed to be the poisoning defense, and it is necessary but not sufficient: provenance tools prove which snapshot trained a model, which lets you trace an incident after the fact. They do not inspect the data for planted triggers before training starts — that job belongs to detectors like the ones in the defense guide above.

When data poisoning fits — and when it doesn't

Common questions about data poisoning

Q: Is a small, low-budget fine-tuning job too small a target to be worth poisoning? A: No — the scale assumption behind that belief has collapsed. As few as 250 documents can poison an LLM fine-tuning set, and fewer than 100 images can compromise a diffusion model’s concept, so a modest training run is not protected by its size. The 2026 threat shift documents the research behind that number.

Q: If my model passes every accuracy benchmark, can I rule out poisoning? A: No — that is precisely the property a well-built backdoor is designed to have. The threat model explains why a poisoned model performs identically to a clean one on every input except the attacker’s trigger, which standard accuracy testing never samples.

Q: Does tracking data provenance alone stop a poisoning attack? A: No — provenance proves lineage, not integrity. You still need detectors running against the data itself; the ART and ML-BOM defense guide lays out how the two layers work together instead of substituting for each other.

Q: My system only calls a locked-down, closed-weight model through RAG — am I still exposed? A: Yes — RAG poisoning targets the retrieval corpus or agent memory store, not the model’s weights, so a hosted model you cannot retrain offers no protection once the corpus itself is compromised.

Part of AI safety and red teaming · closest neighbour: red teaming for AI. New to AI security from a software background? Start with the story: AI Safety Testing for Developers: What Maps and What Breaks.