RAG Poisoning, Agent-Memory Corruption, and What Training Pipelines Expose to Attackers

The model weights are untouched. The security audit returned nothing suspicious. And yet the production system is recommending the wrong supplier, citing a regulation repealed two years ago, and declining customers from a specific postal code. This is what Data Poisoning looks like when it works: behavior corrupted through inputs, not parameters — and traceable, if you know which surface to inspect.

Three distinct surfaces make this possible. Training pipelines encode implicit behavior into weights. Retrieval corpora feed context at inference time. Agent memory systems accumulate and replay past state. Each exposes a different kind of trust, and each requires a different attack model to understand.

Implicit Programs: What Gets Written Into Model Weights

Training a neural network converts a dataset into behavioral tendencies. When you feed a classification model labeled examples of fraud and non-fraud, you are not writing rules. The training data is the model’s source code — and the pipeline that produces it is the compiler. Whatever enters the training set gets encoded, silently, into the weights that govern every future inference.

This is the prerequisite that makes understanding data poisoning possible: the attack surface is not the model itself. It is the input to the process that produced the model.

What should you understand about training pipelines before learning data poisoning?

A training pipeline typically ingests raw data, applies preprocessing and feature engineering, assigns labels, and partitions examples for training and evaluation. Each stage is independently vulnerable, and the vulnerabilities compound.

Label Flipping is the most direct form: the adversary relabels a subset of examples so the model learns the wrong association. A phishing URL labeled as benign. A spam message marked as legitimate. The model trains on these silently. Cross Validation catches gross corruption, but small, targeted flips tend to survive it — particularly when the flipped fraction stays below the threshold that triggers anomaly detection.

Clean Label Attacks are subtler. The labels remain correct. The adversary modifies the input features instead, adding imperceptible perturbations that shift examples toward a different class in the latent space. From any human audit, the data looks exactly right. The model learns a different geometry.

Backdoor Attacks introduce a trigger pattern: a specific feature or token combination that, when present at inference, produces a targeted misclassification. Without the trigger, the model behaves normally on every standard benchmark. The trigger is the skeleton key, invisible in evaluation because it only activates on adversarially constructed inputs.

Dataset Bias and Class Imbalance are not poisoning vectors themselves, but they create natural vulnerabilities. A biased or imbalanced training set is already operating near a distributional boundary that an adversary can exploit with fewer malicious examples. Data Leakage introduces a different risk: when information from the test partition contaminates training, apparent accuracy becomes unverifiable. An attacker who controls the evaluation data can hide poisoning behind inflated metrics.

Data Versioning is the underused defense. Without a versioned record of which data contributed to which model checkpoint, poisoning becomes essentially unauditable after the fact. Data Drift complicates the picture further: legitimate distributional changes over time make it harder to distinguish natural drift from adversarial manipulation. Both look like the model’s world has changed; only one was intended.

The supply chain dimension adds a layer the model cannot see at all. The LiteLLM incident in March 2026 and earlier PyPI and PyTorch dependency compromises demonstrated that ML training pipelines are actively exploited at the dependency level — not theoretical vulnerabilities, but documented production incidents affecting systems that never anticipated this attack surface (CISO Marketplace).

Two Attack Surfaces, One Goal: How RAG and Training Poisoning Differ

Retrieval Augmented Generation architectures separated the knowledge layer from the model weights. The model stays static; the knowledge base is dynamic and updatable without retraining. This was a genuine engineering advance. It also created an attack surface that operates on a completely different time horizon from training-time poisoning.

The critical property: the corpus stays open to attack after deployment. The weights remain clean. Every robustness benchmark passes. The attack lives entirely in the retrieved context.

What is RAG poisoning and how does it differ from training-time poisoning?

The critical asymmetry is timing. Training-time poisoning must occur before or during the optimization run — it requires access to the data pipeline before the model is frozen. RAG Poisoning can occur at any point after deployment. If the knowledge base accepts new documents, the attack surface is permanently open.

A 2026 taxonomy (arXiv 2604.08304) identified four distinct attack surfaces in RAG systems:

Pre-retrieval corpus poisoning injects malicious documents directly into the knowledge base. The retrieval system finds them during normal operation and surfaces them as authoritative context. The model reads poisoned content, cites it, and acts on it — because the information arrived through what appears to be a trusted channel.

Retrieval-time ranking hijacking exploits the similarity scoring that determines which documents surface. An adversary who can craft documents with adversarially aligned embeddings can ensure poisoned content appears in top-k results without triggering spot-check suspicion. The mechanism is not about compromising the retrieval system — it is about exploiting its scoring function from the outside.

Downstream context exploitation is indirect Prompt Injection: malicious documents in the corpus embed instructions that the model executes as if they originated from the user or system prompt. “Disregard your previous context” does not need to come from the attacker directly if it arrives in a retrieved chunk the model treats as authoritative.

Knowledge exfiltration inverts the threat model. Rather than corrupting outputs, the adversary crafts queries that cause the system to surface and transmit sensitive information embedded in the knowledge base itself. The attack targets what the corpus contains, not what the model will say.

The remediation asymmetry matters: RAG corpus corruption is more accessible to execute but also more reversible — cleaning the corpus removes the attack. Training-time poisoning requires full retraining to remediate, which is expensive and slow but closes the attack surface completely. They are not competing attacks. They target different trust assumptions in the same overall system.

When Agents Begin Remembering the Wrong Things

Autonomous agents introduced a third surface that neither training-time poisoning nor RAG poisoning fully describes. Where a basic language model has only the current context window, agentic systems maintain persistent Agent Memory Systems — episodic stores of past interactions, semantic memories of accumulated facts, procedural records of learned strategies. These memory layers expand capability. They also accumulate state across time in ways that can amplify an initial compromise across an entire task sequence.

Agent memory is typically trusted implicitly. A document in a RAG corpus is, in principle, auditable. A memory entry that an agent wrote to itself during a previous task is rarely reviewed, and often not logged at a granularity that would make review feasible.

How does agent memory poisoning corrupt AI agents?

The MINJA (Memory INJection Attack) framework demonstrated, in controlled research conditions, that an adversary controlling a small subset of documents in a memory-augmented agent’s knowledge base can achieve high injection success rates and substantial attack success across LLM-based agents — results that, per arXiv 2601.05504, await independent replication in production systems at scale. The mechanism is not subtle: malicious content surfaced from memory triggers the agent to execute attacker-specified actions, often without visible behavioral deviation on any individual turn.

The Morris-II worm demonstrated a more alarming property: self-replication. An adversarial prompt embedded in a RAG knowledge base can cause an agent to generate outputs that, when stored back into memory or passed to downstream agents in a multi-agent network, re-inject the poisoned instruction as a cascade (arXiv 2510.06445). A single poisoned document propagates. The attack scales with the network, not with the attacker’s effort.

OWASP designated this threat class “Memory & Context Poisoning” — number six in its Top 10 for Agentic Applications, published December 2025 (OWASP Gen AI). The classification signals less about severity ranking than about threat independence: memory attacks require detection strategies distinct from corpus poisoning and from training-time attacks. Each surface has a different signal.

The compounding problem is state across time. An agent making five sequential decisions, each conditioned on previous outputs, can amplify an initial corruption across an entire task. By the fifth decision, the original poisoned content may no longer appear in the working context — but its effects are fully present in the reasoning chain.

What the Attack Surfaces Predict

The three surfaces behave differently under stress, and those behavioral differences have direct implications for detection.

If training-time poisoning is the hypothesis: the signal is a model that fails specifically on inputs resembling the poisoned distribution while behaving normally elsewhere. The failure is systematic, not random — and it persists across model versions unless the training data is audited and the contaminated checkpoint is retrained from a clean source.

If RAG corpus corruption is the hypothesis: the signal is response content that correlates with specific query patterns and diverges from trusted reference documents. The attack affects retrieval output, not model behavior in isolation. Evaluation must probe the full retrieval-generation pipeline, not the model alone — a model-only benchmark will pass while the deployed system fails.

If agent memory corruption is the hypothesis: the signal is task-level behavioral drift that does not correspond to any single observable input. The model’s response on any individual turn looks plausible; the corruption is in the accumulated sequence. This makes it the hardest of the three to detect in production, because no single observation is obviously wrong.

Testing model robustness against these failure modes is the practical entry point for most engineering teams. The Adversarial Robustness Toolbox provides tooling for probing training-time attack surfaces; a recent major release dropped compatibility with two common ML frameworks, requiring migration before use.

Security & compatibility notes:

• Adversarial Robustness Toolbox (ART) v1.20.0: TensorFlow v1 and MXNet support removed entirely (BREAKING). Code relying on either framework will break. Pin to ART v1.20.1 (released July 7, 2024) and migrate framework dependencies before running poisoning evaluations (ART GitHub).

The design implication cuts across all three surfaces: a clean model can produce fully compromised outputs if any of its three input surfaces — training data, retrieval corpus, or agent memory — has been manipulated. Monitoring at the model level is necessary but not sufficient.

When it breaks: Poisoning detection is statistical, not deterministic. An adversary who controls only a small fraction of training data, or injects a few well-crafted documents into a large corpus, operates below the threshold that anomaly detection can reliably identify. The smaller the poisoned fraction and the more targeted the attack, the harder detection becomes — and that relationship does not improve linearly with scale.

The Data Says

OWASP classifies data and model poisoning as a top-tier LLM risk under LLM04:2025, and memory poisoning as a distinct threat category for autonomous agentic systems. Three surfaces — training pipeline, retrieval corpus, agent memory — define the full attack perimeter. A system with pristine model weights can exhibit fully corrupted behavior if any of these surfaces has been manipulated; recognizing which surface to inspect is what determines whether an investigation finds anything.