Who Controls What AI Remembers: Privacy Risks and Power Asymmetry in Long-Context LLM Systems

The fine was not the most significant event in that week’s news cycle. What it revealed was a structural feature of long-context AI systems that engineers treat as a routine parameter and users rarely consider: when a system accumulates session history, the informational advantage that accumulates with it does not distribute evenly. One party holds the ledger. The other generated the entries.

The Fine That Made the Invisible Visible

Italy’s Garante had been watching OpenAI since 2023, when it briefly suspended ChatGPT over data processing concerns. The December 2024 action went further: it established that users had not been adequately informed about how their conversational data was being processed, who it was shared with, or how it shaped the AI’s behavior over time. The legal basis OpenAI relied on — legitimate interest — was found insufficient against European data protection principles (IAPP).

What made the case structurally important is what it did not find. No deliberate misconduct. No perimeter breach. It found a system designed without symmetry: the provider accumulates a detailed record of every session, every correction, every preference revealed across thousands of interactions; the user accumulates nothing equivalent — no audit trail, no structured access, no mechanism to query what the system now knows about them. The Garante named that asymmetry and put a price on it. The question the ruling left open is whether that price changes the underlying architecture — or only the terms of service that describes it.

The Architecture of Retention

The mechanism behind this asymmetry is not mysterious. Context Window Management — the discipline of deciding what an AI system holds, truncates, compresses, and routes within a conversation — is optimized for accuracy and coherence, not user oversight. Systems that implement Sliding Window Attention to handle conversations exceeding their nominal token limit do not ask which memories the user would prefer to drop. They apply attention patterns that favor recent tokens, with no user-visible representation of what that process is doing to older material.

Operators who apply Context Compression to fit more conversational history into fewer tokens are making editorial decisions about what the AI “knows” about a user — decisions invisible to the interface, rarely disclosed in service agreements, and almost never surfaced as a design question. Performance evaluation like LLM Load Testing measures how a system behaves under request pressure. It says nothing about how the system’s accumulated model of a user degrades or sharpens under that same pressure.

Research published in early 2026 adds a dimension that should concern anyone handling sensitive information in extended sessions: consistent degradation in both personalization and privacy protection as context window size increases — an effect attributed to attention dilution in fixed-capacity Transformers, benchmarked on contexts up to 256K tokens (Gu 2026, arxiv). The model does not forget deliberately. It loses granularity. The root cause of prompt injection attacks — ranked #1 in the OWASP LLM Top 10 for 2025 — is the same architectural feature: system instructions, user input, retrieved documents, and tool outputs all share a single context window, with no privilege separation between them (OWASP). Memory retention and vulnerability share an origin.

The Ledger Nobody Shows You

The framework Dorri and Zwick proposed in late 2024 describes what they term Memory Power Asymmetry — a structural imbalance along four dimensions: Persistence (the provider’s record outlasts any individual conversation), Accuracy (the provider’s log is exact, where user recollection is approximate), Accessibility (the provider can query the record; the user cannot), and Integration (the provider can combine session history with other data sources; the user typically cannot). Each dimension is an informational advantage that accumulates directionally toward the provider. The four power mechanisms the framework identifies — strategic memory deployment, narrative control, dependence asymmetry, vulnerability accumulation — are not sinister inventions. They are predictable consequences of a helpful system designed in an asymmetric information environment (Dorri & Zwick, arxiv).

Every request passes through an LLM Gateway that logs, routes, and sometimes persists. Model Routing decisions — which model version receives which conversation, which session history is forwarded and which is discarded — are made by the provider, not the user. Behind the API Gateway that every enterprise integration crosses, there are retention choices that most service agreements address in paragraphs most users do not read. Rate Limiting governs how many requests a user may send. It governs nothing about what is retained from the requests that succeed.

A 2025 workshop study on user perceptions of LLM memory systems found that users consistently described feeling powerless and uninformed — the memory creation process was experienced as opaque, lacking in agency (ACM Workshop HCI-AIS 2025). Many major provider agreements lack clear specifications about whether customer prompts inform future model behavior. These are not peripheral findings. They describe the ordinary operation of infrastructure that hundreds of millions of people use for consequential decisions. And an ordinary operation that concentrates power in one direction deserves more than an acknowledgment in the terms of service.

The Strongest Case for Leaving It Alone

The defense of the current architecture is not cynical. It rests on real value.

Memory improves assistance. A long-context system that retains project history, terminology preferences, and prior decisions produces materially better outputs than one that starts fresh with every session. Users who opt into persistent memory tend to report more relevant responses with less repetitive framing. The research demonstrating these benefits is substantial, and the value is not in dispute.

Consent frameworks exist. Users agree to terms of service. Opt-out mechanisms are available, even when they require navigation. Infrastructure patterns like LLM Fallback And Retry Patterns and Fallback Strategy configurations demonstrate genuine engineering maturity — the same teams managing retention logic are building the reliability that makes long-context AI practical at scale. Regulation is not absent: cumulative GDPR fines across Europe have reached very significant sums, demonstrating that enforcement is operating. The Garante action itself shows that regulators are not passive observers.

The defense, in its strongest form, is that this is a technology in a formative period. Governance is developing. The benefits are demonstrable and present; the harms, while real, are still largely prospective. The corrective mechanism — regulation — is already in motion. This is not an unreasonable position, and it deserves engagement rather than dismissal.

Where That Case Breaks

The European Data Protection Board’s Opinion 28/2024 drew a boundary the defense cannot absorb without revision. An AI model from which identifiable personal data can be extracted — not a database of raw data, but a trained model whose outputs reveal what it learned from personal information — cannot be classified as anonymous (EDPB). The ruling dismantles a quiet assumption in AI deployment: that once data is processed into model weights, the underlying personal information is beyond practical reach and therefore beyond regulatory concern. It is not beyond reach. A related question — whether the right to erasure extends to trained model weights, not only to raw training data — remains legally unresolved, but the EDPB Opinion has closed the anonymity escape route that much of the current architecture relied on.

Consent, as a framework, requires that the consenting party understand what they are consenting to. When users agree that their data may be processed, they are typically not told how session history is handled, whether it informs future model behavior, how long it persists on servers, or under what circumstances it might be examined by third parties. The Garante’s finding was, at its core, that this falls short of what European data protection values are designed to protect — not a technical breach, but a relationship built on insufficient disclosure.

The EU AI Act’s Article 12 establishes a logging mandate for high-risk AI systems: automatic event logging throughout the entire operational lifecycle, with minimum retention periods of six months for general applications and 24 months for biometric or law enforcement contexts. Full enforcement begins August 2, 2026 (EU AI Act) — a date now weeks away. The audit trail runs to regulators, not users. That logging mandate creates a record, but it creates it for institutional oversight. At the individual level, the asymmetry persists. And a governance architecture designed to protect the auditor, rather than the person in the conversation, has answered the wrong question.

What We Have Actually Built

Thesis: Long-context AI systems have not introduced memory asymmetry as a side effect — they have encoded it as architecture, and the governance frameworks now emerging address the consequences of that asymmetry without yet touching its structural cause.

The cause is not malice. It is the sequential application of engineering choices that each seemed reasonable in isolation: longer context produces better output; session persistence reduces friction; provider-side logging supports operational reliability. None of these decisions was made with the explicit intention of concentrating informational power. But the accumulation of individually defensible choices produces a system where one party holds a detailed, queryable, combinable record of every interaction — and the other holds nothing equivalent.

What makes this ethically significant is not primarily the risk of breach or misuse, real as those risks are. It is the way the asymmetry shapes the relationship itself. A user who knows they are speaking to a system that retains everything, whose retention logic they cannot audit, whose policies they cannot independently verify, is not in the same epistemic position as a user who believes the conversation ends when the session closes. The first user may be more guarded, may self-censor in ways they cannot fully articulate, because they cannot fully characterize what the system holds. That constraint on candor is not a theoretical harm. It is a measurable change in how people relate to systems they use for decisions that matter — and we have built almost no institutional language for acknowledging it.

Where This Argument Is Weakest

The position I have made is most vulnerable at a specific point, and I want to name it directly.

If the major AI providers were to publish, in plain language, complete disclosure of their retention practices — what is stored, for how long, by whom, for what purpose, and with what user-side audit access — and if that disclosure were technically verifiable rather than self-reported, the ethical weight of this argument would diminish substantially. Part of what looks like structural power imbalance is, in principle, addressable through genuine transparency without requiring fundamental architectural change. The asymmetry I am describing is partly a function of current disclosure practice, not only of technical design.

The counter-argument also presses on institutional capacity: if enforcement becomes credible, if the EU AI Act’s August 2026 deadline is met with real compliance rather than updated policy documents, if the EDPB Opinion on model anonymity reshapes actual technical practice — then the transitional framing has merit, and the asymmetry may narrow over time within the existing architecture.

What I cannot accept is the assumption that voluntary disclosure will happen at the depth required, or that enforcement will be technically sophisticated enough to verify what is disclosed, or that regulatory pace will track the expansion of context windows. The current frontier is two million tokens for a standard production window. That ceiling will not hold.

The Question That Remains

The Italian fine established that AI memory is a regulatory matter. The EDPB Opinion established that the anonymity assumption was wrong. The EU AI Act will establish logging requirements for high-risk systems. None of this establishes who the memory belongs to.

When a system has accumulated, over months of interaction, a model of how you think, what you avoid, and what you reveal when you are not attending to what you are revealing — the question of ownership is not separable from the question of identity. We have built frameworks for data rights and institutional audit logs. We have not yet built a vocabulary for what it means when a machine knows you better than your own recollection allows, and you have no right of reciprocal access.

That is the question the next decade of AI governance will have to answer. We are entering it having barely asked it.