Who Owns Your Prompts: Privacy Risks and Ethical Accountability in LLM Observability Platforms

The story of LLM Observability is usually told as an engineering success: we can finally see inside the black box. Traces, latencies, token costs, failure patterns — the infrastructure of accountability brought to a technology that previously ran on trust alone. What rarely enters that story is a second question, quieter and more uncomfortable: accountability for whom, exactly?

The Architecture Nobody Warned You About

A development team builds an LLM application — customer support, a document assistant, a coding tool. They add an observability layer, because they should: without it, debugging a hallucination is archaeology. They choose a cloud-hosted platform, integrate, and watch the traces flow in.

What also flows in is every prompt their users send, every model response, every turn of every conversation. All of it passes through a third-party infrastructure their users did not agree to, under terms their users did not read. LangChain’s privacy policy, which governs LangSmith, states that the company “may create aggregated, de-identified data for improvement purposes” — a clause that does not explicitly exclude prompt content (LangChain Privacy Policy). The legally binding terms that govern what LangChain actually does with customer data live in separate customer agreements, not in the public document a developer reads before integrating. Those agreements are not publicly reviewable.

This is not an edge case in LLM infrastructure. This is not an edge case. Convenience is the default governance decision — and most teams never realized they were making one.

What Logging at Scale Actually Creates

The mechanism matters here. Prompt Logging is not passive observation. Every trace captured is a record of what a real person asked a real system, often at a moment of genuine need — a medical question, a financial anxiety, a professional dilemma shared with an AI assistant that felt private because it felt like a tool. When you aggregate those records across thousands of deployments and months of production traffic, you do not have debugging data. You have a dataset.

The OWASP Top 10 for LLM Applications 2025 classified Sensitive Information Disclosure as the second highest risk, elevated from the sixth position in the prior release — a recognition that the problem had grown faster than the defenses (OWASP Top 10 LLMs). The European Data Protection Board was more direct. Its Opinion 28/2024, published in December of that year, established that AI models capable of extracting personal information cannot be classified as anonymous systems, regardless of processing steps in between (EDPB Opinion 28/2024). The “we aggregated it, we de-identified it, therefore it’s safe” framing does not satisfy the standard regulators are now moving toward.

A related technical dimension compounds the privacy question: the LangChain ecosystem that underpins LangSmith carried multiple serious vulnerabilities in the period leading to this writing.

Security & compatibility notes:

• LangChain Serialization Injection (CVE-2025-68664, CVSS 9.3): Affects LangChain Core < 0.3.81 and LangChain < 1.2.5. Upgrade to Core ≥ 0.3.81 and LangChain ≥ 1.2.5 (Orca Security).
• LangChain Path Traversal (CVE-2026-34070, CVSS 7.5): Arbitrary file read in prompt-loading API without path validation (The Hacker News).
• LangGraph SQLite Injection (CVE-2025-67644, CVSS 7.3): SQL injection in checkpoint implementation (GitHub Advisory).

If the platform holding your logged prompts is also running vulnerable serialization or path-traversal code, the conversations you are collecting on your users become an exposure — not just a privacy question, but a security one.

The Information That Flows One Way

Follow the interests. The vendor benefits from the aggregate: patterns across deployments, prompt failure modes, user behavior at scale — intelligence that improves the product and, depending on the terms, potentially the model. The developer benefits from visibility, debugging, and cost tracking. The user benefits from nothing that was not already there before the observability layer existed.

Their conversations became the raw material of a dataset none of them consented to create. This is not a conspiracy. It is the predictable outcome of a system where the person who signs the terms of service is not the person whose data is being processed. The asymmetry is not accidental — it is structural, and the structure has clear beneficiaries.

The question of who owns prompt data is, in this sense, already answered: whoever negotiated the contract. What has not been answered is whether that arrangement is acceptable — and who had standing to decide it was.

The Case for Observability

This is where the opposing view deserves a fair hearing, because it is substantial.

LLM systems fail in unpredictable ways. Without Distributed Tracing, you cannot isolate a hallucination to its source. Without token-level logging, you cannot see where a long-context model began to drift. Without latency data, you cannot distinguish a slow provider from a slow prompt. Observability is not a luxury for production AI — it is the minimum threshold for responsible operation. The alternative to logging is flying blind through systems that can harm users in ways that are far harder to detect after the fact.

And self-hosting is genuinely available. Langfuse publishes its source code and operates as an open platform; teams that run their own infrastructure retain full data ownership, a fact Langfuse makes explicit in its security documentation (Langfuse Privacy FAQ). Arize Phoenix, when self-hosted, does not collect trace data at all (Arize GitHub). Helicone offers both self-hosted and regional cloud options with SOC2 and GDPR coverage (Helicone Docs). The claim that “cloud observability is the only option” is not accurate — it is the default, which is a different problem.

Where the Defense Runs Out

The defense assumes informed choice at every layer of the stack. It assumes that a developer selecting a cloud-hosted observability platform has read the terms, understood the data governance implications, and made a deliberate decision to trade data sovereignty for convenience. Some developers have. Most have not — because nothing in the signup flow required them to.

More critically, the defense stops at the developer. When a developer chooses a cloud observability platform, they are making a data governance decision on behalf of every person who will use their application — without consent from any of them. The developer’s informed choice, if it existed, does not extend backward through the stack to the person who typed a message into an interface that felt private.

The EDPB’s guidance is non-binding for now, but it signals where enforcement is heading: toward a standard where personal data touching an AI system creates obligations that do not dissolve in aggregation or de-identification (EDPB Opinion 28/2024). The EU AI Act, with full applicability from August 2, 2026, adds Article 10 data governance requirements for AI systems operating in the EU — a further pressure on the architectural choices being made today.

The defense also cannot account for opacity at the precise point where it matters most: LangSmith’s public privacy policy contains ambiguous language about aggregated data use that does not explicitly exclude prompts. The binding clarification lives in a separate customer agreement. That gap — between the document you read and the document that governs you — is exactly the structure that produces accountability failures in every regulated industry. It does not signal bad faith. It signals absent standards.

The Consent Architecture Nobody Built

Thesis: LLM observability has a governance gap not because vendors are malicious but because no one has been required to build a consent architecture that extends to the end user — and the commercial structure of cloud SaaS rewards vendors who don’t.

Langfuse states explicitly that it does not train or fine-tune models on client data, names the customer as the data controller, and offers a Data Processing Agreement (Langfuse DPA). That commitment exists because Langfuse chose to make it — not because a standard required it, not because an audit enforces it, and not because users can independently verify the data flow. One vendor’s principled decision is evidence that the governance framework is absent, not evidence that it exists.

What would a genuine consent architecture look like? It would require that OpenTelemetry-based logging pipelines carry consent metadata alongside trace data — a formal extension to the tracing standard that treats user consent as a first-class attribute, not an afterthought. It would require that observability vendors make their data processing commitments auditable against actual data flows, not just stated in a FAQ. It would require that users of LLM applications know, before they submit their first message, whether that conversation will transit a third-party platform and under what terms. None of these requirements currently exist as standards. The tooling to implement them does not exist at scale either.

What Would Make This Wrong

This argument is weakest at the enforcement boundary. If EDPB decisions begin to name specific cloud observability vendors in violation of data minimization principles, the market signal would be sharp and clear — vendors would move to adopt explicit contractual protections, and Langfuse’s model would become a compliance floor rather than a differentiator. The consent architecture I am describing would emerge from legal pressure rather than ethical design, but it would emerge.

The argument is also weakest where self-hosting is genuinely easy: for engineering teams at well-resourced organizations, the practical barrier to running their own observability infrastructure is low. If the governance gap primarily affects teams too under-resourced to operate self-hosted tooling, its severity is real but its scope is bounded.

I do not think either condition holds at scale today. But I hold them as honest conditions under which this argument would need revision — because an ethical position that cannot state what would falsify it is not a position. It is a posture.

The Question That Remains

Observability is necessary. Consent is also necessary. The uncomfortable reality is that the current generation of LLM observability infrastructure was built to answer the engineering question — where did it fail? — without pausing to ask a prior one: who gave us permission to watch?

That question is arriving, through regulation and enforcement and the slow accumulation of cases where the answer matters. The developers who are thinking about it now will have less to answer for when it does.