Data Poisoning

Every other data-borne failure this theme catalogs — leakage, bias, imbalance, drift — is something that happened to the data. Data poisoning is something someone did to it, and that difference reshapes the response: an incident here is a supply-chain compromise, not a modeling mistake, with an audit trail regulators will eventually ask to see. That is why this topic sits at the deliberate-attacker edge of AI safety and red teaming — the practice closest in spirit to red teaming, and the one most other data-quality checks are not built to catch.

The data poisoning reading path: threat model first, defense last

Start with the threat model itself: what corrupted training data does to model behavior explains why a poisoned model can pass every accuracy check and still carry someone else’s intent inside it. From there, backdoor triggers, clean-label, and label-flipping shows exactly why these attack types slip past manual review and standard data-cleaning pipelines — the mechanism any defense has to account for. RAG poisoning and agent-memory corruption then extends the same logic to systems that never retrain at all: a corrupted retrieval corpus or agent memory store changes behavior without touching a single model weight.

Once the threat model is clear, the ART, data provenance, and ML-BOM defense guide turns it into a build spec — provenance tracking, detector layering, and the compliance paperwork EU AI Act Article 11 starts requiring in August 2026. For the state of the actual threat, the 2026 shift from Nightshade to constant-sample attacks documents how the volume assumption collapsed — and closes with the ethics of deliberately poisoning training data as self-defense, worth reading before you decide whether a tool like Nightshade is a shield or an attack inside someone else’s threat model.

How data poisoning differs from red teaming, safety scoring, and data provenance

Three practices get pulled into the same conversation as poisoning, and each answers a different question.

Red teaming probes a model at inference time, hunting for prompts that make it misbehave right now; poisoning is planted at training time and can lie dormant through months of clean red-team runs before its trigger ever appears in a test set. A red team that never happens to type the trigger phrase will certify a backdoored model as safe.

Toxicity and safety evaluation scores what a model outputs, continuously, against a harm taxonomy — but a poisoned model’s normal outputs are exactly that: normal. Guard models grade behavior, not the data that produced it, so they have nothing to flag until the attack condition fires.

Data versioning is often assumed to be the poisoning defense, and it is necessary but not sufficient: provenance tools prove which snapshot trained a model, which lets you trace an incident after the fact. They do not inspect the data for planted triggers before training starts — that job belongs to detectors like the ones in the defense guide above.

Common questions about data poisoning

Q: Is a small, low-budget fine-tuning job too small a target to be worth poisoning? A: No — the scale assumption behind that belief has collapsed. As few as 250 documents can poison an LLM fine-tuning set, and fewer than 100 images can compromise a diffusion model’s concept, so a modest training run is not protected by its size. The 2026 threat shift documents the research behind that number.

Q: If my model passes every accuracy benchmark, can I rule out poisoning? A: No — that is precisely the property a well-built backdoor is designed to have. The threat model explains why a poisoned model performs identically to a clean one on every input except the attacker’s trigger, which standard accuracy testing never samples.

Q: Does tracking data provenance alone stop a poisoning attack? A: No — provenance proves lineage, not integrity. You still need detectors running against the data itself; the ART and ML-BOM defense guide lays out how the two layers work together instead of substituting for each other.

Q: My system only calls a locked-down, closed-weight model through RAG — am I still exposed? A: Yes — RAG poisoning targets the retrieval corpus or agent memory store, not the model’s weights, so a hosted model you cannot retrain offers no protection once the corpus itself is compromised.

Part of AI safety and red teaming · closest neighbour: red teaming for AI. New to AI security from a software background? Start with the story: AI Safety Testing for Developers: What Maps and What Breaks.