Cohen's Kappa

Cohen’s Kappa (κ) is a statistical coefficient that measures how much two raters agree beyond random chance, expressed as a score from −1 to 1.

What It Is

When two people score the same AI outputs — rating model responses as helpful or not, safe or unsafe, accurate or inaccurate — there is a hidden flaw in comparing their results. Two raters might each assign positive labels to 70% of responses while agreeing on entirely different individual cases. They appear consistent; they are not.

Cohen’s Kappa corrects for this by accounting for chance agreement. Developed by psychologist Jacob Cohen in 1960, it asks: given how often each rater uses each label, how often would they coincidentally agree through random selection alone? The formula subtracts that expected baseline from observed agreement, then normalizes. According to Galileo, κ = (Pᵒ − Pₑ) / (1 − Pₑ), where Pᵒ is observed agreement and Pₑ is chance-level agreement.

A concrete example shows why this matters: imagine two critics who each give positive reviews to 90% of films. By chance alone, they would agree on roughly 82% of films (90% × 90% positive plus 10% × 10% negative). An 88% actual agreement rate sounds strong, but it produces a kappa around 0.33 — in the fair range, far from the near-perfect picture the raw percentage implies.

According to NIH PMC, kappa ranges from −1 (systematic disagreement) to 1 (perfect agreement), with 0 indicating chance-level performance. Standard thresholds: above 0.81 is almost perfect, 0.61–0.80 is substantial, 0.41–0.60 is moderate, 0.21–0.40 is fair, and below 0.21 is slight.

One important constraint: Cohen’s Kappa is defined for exactly two raters. When a study involves three or more annotators, Fleiss’ Kappa is the correct extension. For datasets with ordinal scales or entries with missing labels, Krippendorff’s Alpha handles those cases more accurately.

How It’s Used in Practice

The most common place to encounter Cohen’s Kappa is in the calibration phase of AI evaluation studies. Before a full annotation run begins — whether for a preference dataset, a human benchmark, or rubric validation — two raters score the same batch of 50–100 items. The resulting kappa score acts as a gate: if it falls in the moderate or lower range, the rubric needs revision before annotation can proceed; if it clears the substantial threshold, the full run can begin.

This gate matters because annotation quality compounds downstream. A dataset built on rater disagreement teaches noise. Whether the output feeds reward model training, LLM-as-a-judge validation, or published benchmark results, unreliable inter-rater agreement corrupts the entire evaluation chain.

Automated judges are now measured by the same standard. According to arXiv (grading scale study), model-based judges can reach kappa scores comparable to human rater pairs on specific benchmarks — a result that validates their use for automated evaluation, but only because kappa gives a concrete, reproducible reliability bar to aim at.

Pro Tip: Run a calibration batch of at least 50 items before committing raters to a full annotation run. Kappa computed on 20 items is too noisy to be meaningful. Annotation tools like Label Studio compute inter-annotator agreement automatically after calibration — check the IAA dashboard before scaling up.

When to Use / When Not

Common Misconception

Myth: If two raters agree on 85% of examples, they are reliable.

Reality: Percentage agreement ignores how often agreement would occur by chance. Two raters who each label 90% of responses as “good” would coincidentally agree on roughly 82% of items. Their kappa might still fall in the fair range. Always compute kappa alongside raw percentage when reporting inter-rater reliability.

One Sentence to Remember

Cohen’s Kappa is the number that tells you whether your raters actually agree or just happen to apply the same labels — compute it before trusting any human evaluation dataset.

FAQ

Q: What kappa score is considered good enough for AI evaluation? A: According to NIH PMC, scores above 0.61 indicate substantial agreement. For most AI annotation tasks, clearing that threshold before full annotation is the accepted standard; above 0.81 is considered almost perfect.

Q: Can Cohen’s Kappa be used for more than two raters? A: No. Cohen’s Kappa applies to exactly two raters. For three or more annotators, use Fleiss’ Kappa. For ordinal scales or datasets with missing labels, Krippendorff’s Alpha is more appropriate.

Q: What does a negative kappa score mean? A: A negative kappa means raters agree less often than chance would predict — systematic disagreement rather than random noise. This almost always signals a broken rubric or fundamentally inconsistent labeling instructions.

Sources

• NIH PMC: Interrater reliability: the kappa statistic - foundational reference for kappa thresholds and statistical interpretation
• Galileo: Enhancing AI Evaluation with Cohen’s Kappa Metric - applied guide for using Cohen’s Kappa in AI evaluation pipelines