What Is Human Evaluation for AI and How Human Raters Score Model Outputs

In 2022, Ouyang et al. ran an experiment that produced a result worth examining carefully. InstructGPT — a 1.3 billion parameter model — was preferred over GPT-3’s 175 billion parameters by human raters in direct comparison, a finding confirmed in arXiv (Ouyang 2022). Automated benchmarks had pointed the other way.

The discrepancy wasn’t noise. It was the first loud demonstration of a structural fact: what humans call “good” and what loss curves measure are different quantities. Conflating them produces miscalibrated models. That gap is the reason human evaluation exists as a discipline at all — not as the slow alternative to automated scoring, but as the thing automated scoring is trying to approximate.

Running that approximation faithfully turns out to be a nontrivial engineering problem.

How Human Raters Create Ground Truth

The instinct is to frame human evaluation as a quality-check step bolted onto the end of model development. That framing misplaces it in the pipeline. Automated metrics — perplexity, BLEU, even LLM as a Judge systems — are not measuring model quality independently. They are approximating human preference using signals that correlate with it. Human evaluation is the measurement being approximated.

Not the baseline. The reference point.

What is human evaluation for AI?

Human evaluation for AI is the practice of having human annotators assess the outputs of LLM systems according to structured criteria. The assessments take three general forms: pairwise comparison (raters choose which of two responses is better), absolute scoring (raters assign a score on a defined scale), and error annotation (raters mark specific failures — hallucinations, refusals, safety violations — in the text itself).

Each form captures different information. Pairwise comparison collapses multidimensional quality into a single preference signal, which is easy to collect at scale but loses information about why one response won. Absolute scoring preserves dimensionality but requires raters to agree on what each score level means — a harder coordination problem. Error annotation is the most granular and the most expensive to run.

What all three share is their relationship to RLHF: the preference labels human raters generate become the training signal for reward models, which in turn shape how language models are fine-tuned to produce outputs humans prefer. The loop runs from human judgment to model behavior and back. The quality of that loop begins here.

How does human evaluation of AI model outputs work?

A human evaluation study begins long before any rater sees a model output. The first decision is what to measure — which capability, along which dimensions, against which baseline. The second is task construction: how to present the model output to raters, how to phrase the comparison question, and how long each rating task should take to limit fatigue.

Raters then work through a queue of tasks — pairwise or individual — inside an annotation platform. Label Studio is one commonly used open-source option; it provides configurable labeling interfaces with built-in support for multi-rater assignments and data quality tracking. Version 1.23.0 is the current stable release (Label Studio PyPI).

Security & compatibility notes:

• Label Studio XSS vulnerability (CVE-2026-22033): Stored XSS in the custom_hotkeys field; affects versions ≤1.22.0. Fix: upgrade to 1.23.0 (released March 13, 2026; NIST NVD).

Each rater’s judgment is then aggregated — not simply averaged, but analyzed for agreement. Agreement analysis is where the measurement becomes defensible or falls apart.

Inside a Well-Designed Evaluation Study

Designing a human evaluation study is a specification problem. The output — a set of preference labels or quality scores — is only as good as the instrument that generated them: the rubric, the rater pool, and the quality control mechanisms built around both. These three components cannot be treated as implementation details.

What are the key parts of a human evaluation study for LLMs?

Rubric design is where most evaluation studies succeed or fail. A well-formed rubric contains four components: a quality dimension (what is being assessed — helpfulness, accuracy, safety, coherence), a scoring scale, decision guidelines distinguishing adjacent score levels, and scored anchor examples that show raters what each level looks like in practice (Appen). The single most important rubric principle: rate one dimension at a time, not a composite overall score (Appen). A composite score obscures which dimension drove the rating, making the data useless for diagnosing model-specific weaknesses.

The choice of scale is less intuitive than it appears. A 0–5 point scale produces the highest agreement between human raters and LLM-as-a-judge systems, compared to binary or finer-grained alternatives (arXiv grading scale). That alignment matters in evaluation pipelines that use both human and automated scoring at different stages.

Rater pool selection follows rubric design. Raters should have domain familiarity appropriate to the task — rating the helpfulness of a clinical explanation requires different expertise than rating code correctness. Assignment logic, blinding (raters should not know which model produced which output), and task duration calibration all affect label quality in ways that accumulate silently over a long study.

Inter Annotator Agreement measurement is the mechanism through which a study proves its own reliability. The standard tools: Cohen's Kappa for two raters, Fleiss’ Kappa for three or more, and Krippendorff’s Alpha when rater pools overlap across data types (Maxim AI). The formula for Cohen’s Kappa is κ = (Pᵒ − Pₑ) / (1 − Pₑ), where Pᵒ is observed agreement and Pₑ is chance agreement (Galileo). A κ above 0.75 is excellent; 0.61–0.80 is substantial; below 0.4 means the rubric needs revision (Galileo).

Quality control mechanisms complete the structure. Gold standard sets — pre-labeled tasks mixed into the rater queue at unpredictable intervals — measure whether individual raters drift from the rubric over time. A typical quality threshold is around 95% accuracy on gold standard items (Maxim AI). Raters who fall below threshold are flagged for retraining or removed from the pool.

What the Kappa Score Is Actually Telling You

The inter-annotator agreement score is frequently treated as a bureaucratic requirement — something to report in a methods section and then move past. That treatment discards diagnostic information.

If κ falls below 0.4, the rubric is ambiguous. Raters are making different judgments because the decision guidelines leave too much interpretation open, the anchor examples don’t cover the cases they’re actually rating, or the quality dimension is itself poorly defined. A low Kappa is a rubric design problem, not a rater quality problem. Replacing raters without revising the rubric reproduces the same low agreement with a different set of people.

If κ is high but model scores cluster tightly around the midpoint, the scale is compressing variation that exists. Raters agree — but on a scale that cannot distinguish the models being compared.

A few if/then predictions that turn the mechanism into active understanding:

• If your annotation task takes raters more than 5–7 minutes per item, expect agreement to degrade as fatigue errors accumulate toward the end of sessions.
• If you add a quality dimension mid-study without new anchor examples, expect a step-change drop in inter-rater agreement on that dimension specifically.
• If you run a pairwise comparison study without randomizing which response appears on the left, positional bias inflates the apparent preference for one model.

The human evaluation pipeline is a measurement system. It has systematic error sources — rater fatigue, positional bias, cultural variation in quality judgments, domain expertise gaps — that operate independently of the models under evaluation. Diagnosing those error sources is as important as analyzing model performance.

When it breaks: Human evaluation degrades predictably when the quality dimension is subjective enough that expert disagreement is not noise but legitimate variation. In domains like creative writing or open-ended philosophical reasoning, inter-annotator agreement stays low even with well-designed rubrics — not because the instrument is broken, but because human quality judgments genuinely diverge on those tasks. When κ cannot be pushed above 0.5 despite rubric revision, the evaluation question may be unanswerable through preference aggregation at all.

The Number Chatbot Arena Calls ELO

The most widely observed human evaluation system in production is Chatbot Arena, which collects pairwise preferences from voluntary users across live model conversations. The score commonly described as “ELO” is actually a Bradley Terry Model — a probabilistic ranking system that estimates the probability of one model defeating another given the full history of matchups, rather than updating scores incrementally after each pair (Presenc AI). The distinction matters: the Bradley-Terry model handles unequal matchup counts more stably than classical ELO Rating, which is the condition a live leaderboard always operates under.

The math is better. The name stuck.

What Chatbot Arena demonstrates at scale is the gap problem in reverse. As of June 2026, the top eight models sit within roughly 55 rating points of each other (Presenc AI) — a spread so narrow that user preference signals become difficult to separate from positional artifacts and question-type confounds. When models converge this closely, larger sample sizes and tighter experimental controls are required to produce signal over noise — the same conclusion human evaluation methodology reaches from first principles.

The relationship between human evaluation and LLM as a Judge systems follows directly from this. GPT-4-class judges match human preferences more than 80% of the time (arXiv Zheng 2023). That figure sounds like permission to replace human raters. But the agreement percentage was measured against human preference data — data collected through human evaluation studies structured exactly as described above. The calibration of the automated judge runs through the human ground truth. You cannot remove the foundation and keep the building.

The same distinction applies to SWE Bench Verified: its “human-verified” subset refers to the task curation process (humans confirmed that issues are solvable from real GitHub repositories), not to human scoring of model outputs. These are different uses of the word “human” in AI evaluation and should not be conflated.

The Data Says

Human evaluation is the measurement source that automated scoring approximates. Its reliability is a function of rubric precision, rater agreement, and quality control discipline — not primarily of the number of raters or the size of the annotation budget. A study with 1,000 raters working from an ambiguous rubric generates noise at scale. A study with 50 raters and a Cohen’s Kappa above 0.75 generates signal that can calibrate a reward model, fine-tune a language model, and anchor an automated judge. The models being evaluated are the last variable to examine when inter-annotator agreement is low.