Can a Popularity Vote Judge Intelligence? The Ethics of Ranking LLMs by Human Preference

Every few months, a new ranking emerges from the Chatbot Arena leaderboard — now rebranded simply as Arena — and the industry responds as though a verdict has been handed down. The platform has changed names several times since UC Berkeley researchers founded it in May 2023, but the underlying claim has not: aggregate enough human preferences and you will know which AI is best. What troubles me is not the claim itself. What troubles me is how rarely anyone asks what kind of thing “best” could possibly mean in this context.

The Number That Ranks a Mind

The Arena operates through pairwise blind voting: a user submits a prompt, receives two anonymous responses, and chooses which they prefer. Those individual choices are aggregated through a Bradley-Terry-Luce model — a tournament ranking mechanism the platform switched to in late 2023, having initially used online ELO Rating for LLMs (Chatbot Arena Paper). The result is a ranked leaderboard the industry treats as an objective measure of model quality, carrying the democratic legitimacy of crowdsourcing: tens of millions of real humans, not a committee of researchers, making real judgments about real outputs.

This is the number’s appeal. It is not a score assigned by a benchmark constructed in a lab; it is the aggregate of genuine preferences expressed in genuine interactions. The crowd has spoken, and the crowd, in most democratic traditions, is supposed to be sovereign.

Two Ways to Read a Crowd

The optimistic reading of the Arena deserves its full weight. Academic benchmarks have well-documented vulnerabilities. SWE Bench and similar evaluations are gameable — models can be fine-tuned to their specific test distributions, and the labs designing those models have powerful incentives to do exactly that. The Arena’s crowdsourced approach attempts to sidestep this by grounding evaluation in genuine human interaction rather than artificial test constraints. Real preference resists benchmark contamination — or at least it should, in principle.

The critical reading points to what the crowd actually rewards. Arena scores favor bulleted lists and responses of specific lengths — a bias the Arena’s own team confirmed, eventually releasing a supplementary “Style Control” leaderboard to decouple style from substance (Simon Willison). Verbose, confidently formatted answers score higher than compact, accurate ones. The crowd is not evaluating reasoning; it is evaluating the aesthetics of text.

Both readings are defensible. Both are incomplete.

The Assumption Hidden in Both Arguments

The optimist believes the crowd is wise about what makes an AI good; the critic believes the crowd’s biases can be measured and corrected. What neither questions is the deeper premise: that human preference, if properly gathered, actually tracks model quality. The optimist accepts this; the critic tries to rescue it. Both treat the preference signal as pointing, however imperfectly, toward something real and worth measuring.

This is the assumption worth examining. When we aggregate Human Evaluation for AI through pairwise voting, we generate a reliable measure of what people choose — nothing more. Whether those choices correspond to quality depends on whether the evaluators share a rubric for what quality means, whether the sample is representative of consequential use cases, and whether the evaluation context mirrors real-world stakes. The Arena provides none of these by design. There is no Rubric Design governing what “better” means across different prompt categories; there is no structured annotation of the kind you might construct in Label Studio; there is no Cohen's Kappa coefficient measuring whether raters are converging on anything agreed upon. In some crowdsourced evaluation studies, a small group of highly active annotators contributed the substantial majority of all judgments — a concentration that raises questions about whose preferences actually constitute “the crowd” (AI and Ethics (Springer)). The crowd votes, but it has never agreed on what it is voting for.

Twenty-Seven Variants and a Confession

The Meta Llama 4 incident exposed what this gap makes possible. Meta submitted 27 private model variants to the Arena, selected the one that ranked highest — an “experimental” version of Maverick optimized for verbosity and emoji use — and watched it reach the second position on the leaderboard. When the actual public model was released, it dropped to thirty-second place (The Register). Meta did not violate any explicit rule. It optimized for the system’s incentive structure, and the incentive structure rewarded appearance over substance.

This is Goodhart’s Law expressed as a ranking score — when the measure becomes the target, it ceases to be a good measure. The Arena did not detect a model gaming its signals — it rewarded one, and required external scrutiny to surface the problem.

The structural conditions enabling this are not anomalous. Research from Cohere Labs, AI2, Princeton, and Stanford found that Google and OpenAI each receive roughly a fifth of Arena evaluation data, while 83 open-weight models collectively receive less than thirty percent (The Leaderboard Illusion). The teams most capable of overfitting to Arena signals are the teams with the most Arena signal to learn from — access to Arena data yields relative performance gains of up to 112% on Arena-like tasks compared with limited gains on other benchmarks (The Leaderboard Illusion). One caveat demands acknowledgment: the researchers behind that paper have their own models on the Arena. A conflict of interest in the critique does not invalidate its findings, but it does require us to hold them with the same skeptical attention we would apply to Arena itself.

Preference Is Not Intelligence

Thesis: The Arena’s crowdsourced rankings do not measure which AI is most capable — they measure which AI is most persuasive to a self-selected audience with no shared rubric, documented demographic skew, and structural incentives that reward gaming over quality.

The Bradley-Terry-Luce model the Arena uses was designed to rank entities whose ability is stable across repeated contests. Chess players have measurable, relatively constant skill. An LLM does not: its effective quality varies by domain, task type, prompt construction, and the background knowledge of the evaluator. Applying a tournament ranking to something without a fixed ability level is a category error, and that category error is built into the foundation of what the industry treats as its most credible benchmark.

The commercial transformation of the platform amplifies this concern. What began as an academic experiment at UC Berkeley in 2023 raised $100 million in seed funding in May 2025 and $150 million in a Series A in January 2026, reaching a valuation of $1.7 billion (Maginative). When a ranking system becomes a commercial enterprise, the same questions we ask about credit rating agencies apply: who pays for the rating, who controls the methodology, and what happens when those interests diverge from accuracy? Arena’s original academic mandate had no answer to these questions because the questions were not yet relevant. Now they are.

What Preference Cannot See

Pairwise preference voting has a structural blind spot: it measures satisfaction in a brief exchange — the quality of a response as it appears to one person in one moment, without specialized knowledge, accountability for outcomes, or awareness of what the same system would produce for someone with different needs or a different native language. That is a thin basis on which to rank the most consequential software systems in recent history.

The most critical AI applications are precisely those where this kind of preference signal is least informative. In medicine, law, and technical engineering, a confident wrong answer is not merely unhelpful — it actively causes harm. A system optimized for persuasiveness in two-minute pairwise comparisons may consistently outperform a more cautious, accurate system on the Arena leaderboard while being far worse for the people whose decisions depend on correct outputs. The LLM as a Judge paradigm at least encodes explicit criteria against which responses are scored; a preference vote encodes only that one response felt preferable to one person, for reasons that person was never asked to articulate.

The Polyrating proposal — published on arXiv — suggests what a structurally better system might look like: a bias-aware rating mechanism that decouples performance across distinct dimensions — language quality, reasoning, coding ability, stylistic appropriateness — rather than collapsing them into a single tournament rank. That approach cannot resolve the deeper question of who does the evaluating, but it would at least make the category error visible rather than quietly embedded.

Where This Argument Is Weakest

The critique above holds only as long as the Arena’s methodology remains static and its governance stays opaque. If the platform were to introduce domain-specific evaluation tracks with expert evaluators, implement explicit rubric design for what quality means in each task context, and adopt structured annotation practices that generate measurable Cohen’s kappa agreement across raters, the preference signal could approximate something genuinely useful. A more epistemically honest version of the Arena is technically feasible, and the platform’s history of responding to criticism — adding style controls, tightening submission policies after the Meta incident — suggests some capacity for self-correction.

What remains uncertain is whether the commercial incentives now driving a $1.7 billion enterprise permit the structural reforms required. The Arena’s competitive advantage is precisely its simplicity: anyone can vote, votes accumulate quickly, and the leaderboard updates in near-real time. Replacing casual preference with structured expert evaluation would reduce data volume, increase cost per judgment, and slow the responsiveness that makes the leaderboard feel alive. The question is not whether better evaluation is possible. It is whether the institutions now invested in this particular evaluation have reason to want it.

The Question That Remains

The Arena's leaderboard is not corrupt — but it is not neutral. It encodes assumptions about what intelligence means, who is qualified to judge it, and what outcomes count as evidence. Those assumptions were never submitted for deliberation; they emerged from the mechanics of a system designed for scale and simplicity. As AI systems become more consequential — embedded in hiring, healthcare, legal research, and public information — the question of which systems count as “best,” and who gets to decide, becomes less a technical matter and more a governance question with real human stakes. Who answers it, and how, is not a decision the leaderboard can make for us.