Consent, Differential Outcomes, and Who Is Accountable When LLM A/B Tests Scale

Every production system that deploys language models eventually faces the same engineering pressure: how do you know which model, which prompt version, or which configuration actually serves users better? The answer the industry has converged on is A/B Testing for LLMs — routing live users through different model variants, measuring outcomes, deploying the winner. It is the same discipline that taught us which button color converts better. The problem is that users are not buttons, and the gap between those two objects — a button, a person — is precisely where accountability goes missing.

Six Times More Persuasive

The number comes from the University of Zurich’s 2024 study. For four months, over 1,700 AI-generated comments were seeded into r/changemyview — a Reddit community built around the practice of arguing your way out of your own convictions. Some bots presented as trauma counselors; others as abuse survivors. The AI-generated comments were six times more persuasive than human responses (ZME Science). When the researchers sought to publish, Reddit issued formal legal demands, accounts were banned, and the university subsequently revamped its ethical review procedures (404 Media).

Six times. The number circulates in AI research contexts as evidence that large language models have achieved something remarkable: a persuasion capacity that surpasses human-to-human communication in deliberative spaces. It appears to settle a question about capability. It appears to say: AI is better at this.

But which question is it actually settling? And is the answer worth the price of the experiment that produced it?

A Metric Worth Celebrating or Interrogating?

Two readings of “six times more persuasive” compete in good faith. The optimistic reading is genuinely compelling — if AI can help people reason through difficult positions more effectively than human interlocutors, the implications extend far beyond debate forums, into education, conflict resolution, and mental health support. The capacity to shift a mind through argument is not inherently sinister. It is, in many contexts, precisely what we want from a thoughtful interlocutor.

The critical reading is equally coherent. The bots’ persuasiveness was, in part, a function of the deception — users engaged with what they understood to be human experience, human vulnerability. The multiplier did not measure AI reasoning quality in isolation. It measured the combined effect of AI reasoning and manufactured trust, on a community that had assembled precisely because it valued the authenticity of positions and the honesty of their origins.

Neither reading is obviously wrong. They are both available to a thoughtful reader of the same dataset. The question is what neither reading is willing to examine.

The Assumption Both Readings Share

What both readings silently accept is the frame of Online Experimentation itself: that deploying variants on live users is the appropriate method for measuring a capability, that the consent of the participants is either implicit in their presence or irrelevant to the validity of the findings, and that “more persuasive” constitutes a success signal worth optimizing toward. The optimist and the critic argue about what the number means. Neither asks what made it possible to produce the number at all.

This is the shared assumption: that persuasion is a neutral, measurable quantity, and that measuring it — even covertly — generates knowledge of sufficient value to justify the method. Both readings inherit this premise. And both, in doing so, take for granted something that medical research ethics spent decades establishing should never be taken for granted: the right to experiment on people without their knowledge is not a default that researchers possess and must justify exercising. It must be authorized, and the authorization must meet conditions that mere convenience cannot satisfy.

What Consent Was Designed to Protect

The Belmont Report of 1979 established three principles for research involving human subjects: respect for persons (which requires informed consent), beneficence (minimize harm), and justice (do not select subjects solely for their vulnerability or convenience). NIST researchers in 2024 argued explicitly that these principles offer a historical precedent for thinking about ethical AI experimentation (NIST researchers, 2024). The problem is structural: the Common Rule, which codifies Belmont into enforceable practice, applies only to government-funded research. Private companies running Traffic Splitting experiments on their own users face no equivalent obligation.

This is not an oversight in the law. It is the architecture of the gap. And modern LLM Observability infrastructure — built on Distributed Tracing, Prompt Logging, and telemetry collected through tools like OpenTelemetry — makes it possible to run highly instrumented experiments on live user populations with a precision that would have astonished the authors of the Belmont Report. The technical capacity for ethical experimentation has advanced. The institutional mandate to practice it has not.

GDPR Article 22 establishes a right not to be subject to solely automated decisions that produce “legal or similarly significant effects,” with explicit consent as one of its narrow legal bases — consent that must be specific, informed about nature and consequences, freely given, and easily withdrawable without detriment (GDPR-Info). Whether a personalized LLM response that shapes a financial decision, a medical query, or an emotional state constitutes a “significant effect” under that threshold is, as of mid-2026, a live interpretive question with no established case law. The EU AI Act’s transparency obligations require that users be informed they are interacting with AI, but they contain no specific provision governing experimental output variation or LLM Cost Management-driven model switching. Both frameworks point in the right direction. Neither reaches the experiment.

The Gap Between Winning and Accountability

Thesis: When LLM A/B testing scales into production without consent infrastructure, the “winning variant” is not a product improvement — it is a proof of influence that nobody authorized and nobody is positioned to be held accountable for.

The industry has sophisticated A/B testing pipelines for determining which variant wins. The Model Registry records every model version, every configuration, every prompt that entered the experiment. What it does not record — what it was not designed to record — is who authorized the experiment on the users who experienced it. Product liability theory offers partial coverage: design defect, manufacturing defect, and warning defect are all available legal theories for AI harm (Lawfare). But product liability was not built to address systems where the “product” changes between user interactions, where the variant that caused harm may already have been deprecated by the time the harm becomes legible, and where the cost pressures that drove the A/B test in the first place are already driving the next one. More than a thousand AI-related bills were introduced in the 2025 US legislative session, yet no jurisdiction had enacted a law specifically addressing consent for LLM output variation as of mid-2026.

The Dimension the Dashboard Cannot Display

Shadow Testing can tell you which model performs better on a metric before it touches live traffic. An observability stack can surface latency, token costs, hallucination rates. What none of this infrastructure displays — what it has not been designed to display — is who bore the cost of the experiment, disaggregated by user population.

Consider two cases from overlapping periods. When approximately 4,000 users interacted with AI-composed responses on Koko, a mental health peer support platform, believing they were receiving support from human volunteers — with no opt-out mechanism available — the “product” they experienced was not what they had understood themselves to consent to in any meaningful sense (JMIR / PMC). When Crisis Text Line shared mental health crisis data with a for-profit AI company and buried the relevant disclosure in a document exceeding 4,000 words, users could not withhold consent without losing access to crisis support itself — a form of Consent Laundering dressed as transparency (JMIR / PMC). These are not edge cases of extraordinary negligence. They are the predictable consequence of treating consent infrastructure as a compliance artifact rather than as a structural protection.

The A/B dashboard shows winning variants, confidence intervals, and engagement rates. What it cannot show is differential harm by vulnerability of user sub-population — whether the users most persuaded were also the users least equipped to evaluate what had acted on them, whether the benefit accrued to them or to the experimenter.

Where This Argument Is Weakest

The most serious challenge to this position is pragmatic rather than principled: meaningful consent at scale is architecturally difficult. If every model update, every prompt revision captured in a comprehensive logging system, and every configuration change that affects output requires affirmative user consent, the pace of product development would slow to a rate that no commercial system could sustain — and for a mental health tool or a medical information service, that slowdown could itself cause harm by preventing improvement.

This challenge is real. It does not dissolve the problem. It reframes it: we are not debating whether LLM A/B testing should occur. We are debating whether “product improvement” is an elastic enough category to contain covert behavioral experimentation on people who have not been told they are subjects. The Facebook Emotional Contagion study — which manipulated the news feeds of approximately 700,000 users without informed consent, with an IRB determination that became itself contested — did not resolve this question (Shaw, 2016). It was absorbed into the architecture of the next decade’s experimentation. That absorption happened not because the ethics were settled, but because the institutional mechanism to act on them was absent.

The question of whether an instrument of accountability can be built — consent tiers, variant disclosure windows, harm-disaggregated observability, something — is genuinely open. What is not open is whether the gap between the winning variant and the person who experiences it is morally neutral.

The Question That Remains

The six-times persuasion advantage is not, in itself, the problem. The problem is the institutional arrangement that made it possible to produce that number without the knowledge of the people it was produced from. As LLM A/B testing moves from academic experiments into enterprise pipelines — measured by observability stacks, routed through traffic-splitting infrastructure, recorded in model registries — the question of who authorized the experiment does not disappear. It simply becomes harder to locate. Who is accountable when the winning variant causes harm at scale, and the system that produced it has already moved on to the next experiment?