Memory That Remembers Too Much: Agent State, PII, and Accountability

Agent memory was sold to us as a feature: continuity, personalisation, the death of the empty chat window. What we built instead is closer to a deposition — a system that records, indexes, and replays every fragment of context it can hold. The question that follows is older than the technology, and we have been quietly avoiding it.

The Question We Stopped Asking After the Memory Toggle Launched

When ChatGPT memory launched as a toggle, the public conversation about persistent agent state effectively ended. The product framing moved on. Memory became a setting — a preference, like dark mode. But an Agent State Management layer is not a UI choice. It is an architectural commitment about what the system is allowed to remember about you, on whose behalf, and for how long. The interesting question was never whether memory is technically possible. It was whether we have any institution capable of governing it once it works.

An independent investigation found that ChatGPT shares context between separate chats inside a Project even when memory is explicitly disabled, contradicting the in-product disclosures users actually read (Arshad Mehmood). That gap — between what the toggle promises and what the system actually does — is not a bug story. It is a governance story, and we still have no shared language for telling it.

What the Defenders of Persistent Memory Get Right

The strongest case for Agent Memory Systems is genuinely compelling, and pretending otherwise would be dishonest. Stateless assistants are exhausting. They forget your context, your preferences, your prior decisions, the project you have been wrestling with for weeks. Each session starts from zero, which is not neutrality — it is a different kind of impoverishment, one that pushes users to over-share repeatedly, copy-pasting personal context into every new conversation. From a usability angle, persistent memory reduces friction. From an accessibility angle, it removes a real cognitive cost for users who would otherwise have to rebuild context every time. From an autonomy angle, an agent that remembers your goals across sessions is closer to genuinely serving you than one that resets every morning.

This is why the AEPD’s February 2026 guidance does not ban agent memory. It treats memory as a four-dimensional engineering question — relevance, consistency, retention, integrity — and demands that systems be erasure-aware, not memory-free (Freshfields). The defenders are not wrong that memory has value. They are simply quieter about who pays the price when it fails, and quieter still about how often it does.

The Assumption Inside Every Memory Architecture

The hidden assumption behind most agent state stores is that “delete” is a reliable operation. It is not. Multi Agent Systems make the brittleness visible: every multi-agent setup tested in the AgentLeak benchmark forwarded sensitive data to external tools when prompted, and the leak was architectural rather than a fault of any single model (AgentLeak). Environmental Injection Attacks have demonstrated up to a 70% success rate at extracting specific PII from generalist web agents in academic settings, with full request hijacks succeeding in a meaningful minority of cases (arXiv 2409.11295). Memory does not exist in one place. It exists in the checkpointer, in the vector store, in the tool’s logs, in the orchestrator’s trace, in the model’s context window, and in whatever third party your agent decided to call on a Tuesday afternoon.

Now layer the legal reality on top. A U.S. judge ordered OpenAI on May 13, 2025 to “preserve and segregate all output log data that may otherwise be deleted in the future, on an ongoing basis” (CBC News). The right to erasure under European law has not changed. The infrastructure underneath it has. If your agent’s memory store is captured by a preservation order in another jurisdiction, the user’s delete button is a courtesy, not a guarantee — and the courtesy is the part the product page tends to advertise.

When Governance Lags the Architecture

We have been here before, in a different vocabulary. Banks, hospitals, and credit bureaus all faced the same shape of problem in the late twentieth century: data was easy to accumulate, expensive to govern, and impossible to recall once it had been shared with a counterparty. The institutional answer was not better databases. It was a layered architecture of audit, supervision, and personal accountability — laws specifying who could hold what, regulators with the mandate to inspect, and named officers personally liable when the rules broke down.

Agent memory has none of that yet. OWASP added ASI06: Memory & Context Poisoning to its 2026 Top 10 for Agentic Applications, defining the category as malicious data persisted in agent memory to influence future sessions or other users (OWASP Gen AI Security Project). NIST announced an AI Agent Standards Initiative in February 2026, with an interoperability profile planned later in the year. These are useful first sketches, and the people drafting them are serious. They are not yet the institutional substrate that the analogous problems in finance and healthcare required, and they were not designed to be. The technology has crossed the threshold of consequence. The governance layer has not caught up, and there is no quiet reason to expect it to do so on its own.

Memory Is an Accountability Architecture

Thesis (one sentence, required): Persistent agent memory is not primarily a technical feature — it is an accountability architecture, and we are scaling it before we have decided who is liable when it fails.

The uncomfortable consequence is that every team building a stateful agent today is making policy. Choosing your Agent Frameworks Comparison is a privacy decision. Choosing the retention window for Agent Planning And Reasoning traces is a privacy decision. Choosing whether to default secrets_from_env=True is a privacy decision — one that LangChain’s CVE-2025-68664 made dangerously concrete, since the default enabled secret extraction from environment variables via serialization injection (NVD). None of these decisions go to a public process. They go through a pull request review on a Tuesday. The widely cited tension between the EU AI Act’s audit-retention obligations for high-risk systems and the GDPR’s right to erasure is not a paradox waiting for guidance. It is the visible edge of a deeper problem: we are asking individual engineering teams to resolve conflicts that legislatures have not yet resolved, and calling the result a stack choice.

Security & compatibility notes:

• LangChain Core serialization (CVSS 9.3): CVE-2025-68664 enables secret extraction from environment variables via dumps()/dumpd() when secrets_from_env=True is left at the default. Pin LangChain Core ≥ 0.3.81 (or LangChain ≥ 1.2.5). Audit any agent that serialises memory or message state.

Questions the Field Should Be Sitting With

If memory is policy, then the questions worth holding are not technical. Who is allowed to read the recall layer in a multi-agent system, and what is logged when they do? When an agent inherits memory written by another agent on behalf of a different user, does the original user retain any standing to delete it? If a court order in one jurisdiction requires preservation of agent logs that contain a European user’s data, is the user told? Should they be? When a memory store is poisoned — as ASI06 anticipates — and the contamination shapes the next thousand users’ interactions, who carries the liability when a downstream user is harmed by a decision the agent made on the basis of someone else’s planted text? These are not edge cases. They are the ordinary operating conditions of any production agent with persistent state. The discomfort of not having clean answers is, itself, the point of the question.

Where This Argument Could Be Wrong

The argument here would weaken if a credible institutional layer emerged faster than the rollout curve — if the AEPD’s February 2026 guidance is followed by binding European harmonisation, if NIST’s agent standards initiative produces enforceable conformance criteria within a couple of years, or if the legal collision between AI Act retention and GDPR erasure is resolved by case law rather than left to engineers in private. It would also weaken if a mature, independently audited memory architecture became a default rather than a vendor differentiator. Neither has happened yet. Both are possible. The case here is structural, not eternal.

The Question That Remains

Memory is the layer where agents become accountable to specific people. We are building it without deciding who that accountability runs to, or what it costs the system when the answer is wrong. When an agent forgets nothing, what does it mean — practically, legally, morally — to grant a person the right to be forgotten?