Demographic Parity vs. Equalized Odds: The Ethics and Accountability of Biased AI Data

We have spent a decade trying to turn fairness into a measurement — something a tool can check and an auditor can sign off. The appeal is obvious: a number feels like it ends the argument. But when a metric settles a moral disagreement by quietly ignoring half of it, the disagreement has not been resolved. It has only been moved somewhere nobody is looking.

One Score, Two Verdicts

The score came from COMPAS, a tool used across American courtrooms to estimate how likely a defendant was to reoffend. It promised to trade the prejudices of tired judges for the steadiness of data, to convert a fraught human decision into something that looked like arithmetic. When ProPublica examined those predictions in 2016, the number stopped looking neutral. The model had learned from records that already carried Dataset Bias — arrests, charges, and convictions shaped by decades of uneven policing — and it handed that history back as a forecast about the future.

A number meant to end an argument had started a much larger one. The question was no longer whether the algorithm worked. It was what “working” had ever meant.

When Both Audits Pass

Northpointe, the company behind the tool, now Equivant, defended it on solid ground: the scores were calibrated within each group, meaning a given score carried the same real probability of reoffending regardless of race. ProPublica stood on ground just as solid: among defendants who never went on to reoffend, Black defendants were flagged as high-risk more often than white ones, a higher false-positive rate. Both findings were true at the same time, drawn from the same data.

This is the collision the whole field keeps rediscovering. Demographic Parity asks only that a model select people from each group at the same rate, a definition formalized by Dwork et al. in 2012. Equalized Odds raises a different bar — equal true-positive and equal false-positive rates across groups, conditional on who actually reoffended, as Hardt, Price and Srebro set out in 2016. Each audit can pass on its own terms while the other quietly fails. If two careful, honest audits of the same model reach opposite verdicts, which one of them was measuring fairness?

What Both Camps Took for Granted

Listen to how the two sides argued, and you hear something they shared. Both believed fairness was a property you could compute — that somewhere in the data sat a correct measurement, and the only task was to locate the right one. Northpointe reached for calibration. ProPublica reached for error-rate balance. Neither paused to ask whether justice was the kind of thing a single statistic could hold.

That shared faith is the quiet assumption underneath nearly every fairness audit produced today: that a good enough metric can adjudicate a question about who deserves what. It is a comforting assumption. It is also the one that breaks.

The Theorem That Closes the Door

In the same year ProPublica published, Kleinberg and his colleagues proved an impossibility theorem: when base rates differ across two groups, calibration and balanced error rates cannot both hold unless the classifier is perfect or the base rates are identical. Chouldechova reached the same wall in 2017 from the criminal-justice side. The incompatibility is not a defect waiting for a patch. It is a theorem.

The same logic traps demographic parity and equalized odds against each other. Equal selection rates and equal error rates pull in opposite directions the moment two groups carry different base rates, and no amount of tuning reconciles them. We know how to fix the failures that live below this one — how to catch contamination with Cross Validation, how to hunt down Data Leakage between training and test sets, how to resample around Class Imbalance. Those are engineering problems, and engineering problems have answers. Open-source toolkits like Aequitas can even measure how far apart the competing fairness definitions sit. What no tool can do is make those definitions agree, because their disagreement is not an error. It is a choice the data refuses to make for us.

A Moral Choice Wearing a Lab Coat

Thesis: The choice between demographic parity and equalized odds is a moral judgment about whose errors we are willing to tolerate, and dressing it as a technical metric lets everyone involved avoid owning the decision.

Each definition carries its own cost, and intellectual honesty means naming both. Demographic parity equalizes who gets selected but ignores the true outcome, which can force a system to treat genuinely different cases as though they were the same. Equalized odds preserves that outcome signal but permits unequal selection rates, which can leave a whole group under-served even when the error rates line up. Neither is the fairer standard in the abstract. Is equalized odds fairer than demographic parity? Only once you have answered fairer for whom, and at whose expense — and that sentence belongs to ethics, not to engineering.

Regulation is beginning to sense the gap. The EU AI Act’s Article 10 asks providers to detect and mitigate bias in the data that trains high-risk systems (EU AI Act). That is a meaningful direction for society to be heading. But a rule that says mitigate bias cannot tell you which fairness you owe, any more than a speed limit can tell you where you are trying to go. When the choice vanishes into a metric, accountability vanishes with it — and a decision nobody admits to making is one nobody can be asked to defend.

The History the Metric Cannot See

Return to where those base rates came from. They differ not because two groups of people are different, but because the records are scarred by everything that happened before the model existed — Selection Bias in who gets stopped and charged, Representation Bias in who appears in the data at all, Measurement Bias in using rearrest as a stand-in for crime when rearrest measures policing at least as much as behavior. Bias is never one thing. It seeps in through who gets counted, who gets recorded, and what the labels were quietly standing in for long before any model touched them.

A fairness metric reads the disparity in the outputs. It cannot read the history that produced the inputs. It sees that the numbers differ; it cannot see that the difference is itself a record of harm. And it certainly cannot measure the thing a defendant actually lives through — the quiet indignity of being handed a number, by a process they are not allowed to inspect, that decides how the system will treat them. The algorithm optimizes for a balanced error rate. What does a balanced error rate know about that? A just society would have to optimize for something the metric was never built to hold.

Where This Argument Could Break

This case rests on one empirical claim, and it is worth marking where it gives way. If the base rates across groups were genuinely equal — if the underlying world were already just — the impossibility would dissolve, the metrics would converge, and the choice between them really would collapse into a technical one. The argument also weakens if you believe a society can specify its fairness criterion fully and in advance, openly choosing the trade-off and binding those who build the systems to it. In that world the metric would not launder a decision; it would carry out one that someone had actually made and could be held to. The position is most fragile for anyone who thinks fairness can be finished — written down once and computed forever after.

The Question That Remains

The math is settled; the meaning is not. We can measure the trade-off with real precision, but precision about a trade-off is not the same as the right to impose it on someone else. So when a model decides who is risky, who is creditworthy, who is seen — who gets to choose which definition of fairness they will live under, and who is simply told the number was objective?