Bias Amplification and the Impossibility Theorem: Why You Can't Fully Debias a Dataset

A team strips every gender label out of a hiring dataset, balances the classes until the spreadsheet looks symmetric, and trains the model. It comes back more skewed than the data they started with. Nothing was misconfigured, no column was mislabeled. The bias they thought they deleted was never living in the field they dropped, and the model did something to what was left.

That last point is the part most debiasing efforts miss. We tend to picture bias as a contaminant: a stain in the data that a good enough cleaning pass will lift out. The phenomenon doesn’t behave that way. It enters through multiple doors, encodes itself redundantly across features, and gets sharpened rather than preserved by the learning process. Understanding why requires separating two different walls you run into, and they are not the same wall.

Bias Is Not a Stain You Can Wash Out

Start with the data itself, before any model touches it. The cleaning metaphor assumes a single source of contamination and a single point of removal. Dataset Bias has neither. It is better understood as several independent inputs arriving through different channels, so closing one channel leaves the others wide open.

Why can’t dataset bias be completely removed from training data?

Bias does not enter a model at one door. It enters at several, and the data is only the first room. A widely cited framework counts seven distinct points of harm across the machine learning life cycle, three of which live in the data stream alone: historical bias, where the world being measured is already skewed; representation bias, where sampling under-counts some groups; and measurement bias, where the feature or label you record is a flawed stand-in for the thing you actually care about (Suresh & Guttag). The other four — learning, aggregation, evaluation, and deployment — arrive after the data is frozen. So “fix the data” cannot, even in principle, reach the majority of them.

The three data-stream sources are themselves separate problems. Selection Bias comes from how examples were chosen, Representation Bias from who is under-sampled relative to the target population, and Measurement Bias from a proxy label that quietly encodes something other than ground truth. Patching one does not touch the next.

Then there is the trap that defeats naive scrubbing. Suppose you delete the protected attribute outright. The signal survives, because it is redundantly encoded across other features — postal code, purchase history, the phrasing of a name, the school in an education field. A model with enough capacity reconstructs the attribute from its correlates and goes on discriminating, now invisibly. Dropping the column does not drop the information.

Not removed. Relocated.

This is also why the standard taxonomies treat bias as structural rather than incidental: the sources sit at different layers and resist a single fix. You can reduce bias. You cannot guarantee its absence by editing a table.

How Models Make Bias Worse Than the Data

If the data problem only replicated existing skew, balancing the dataset would at least bound the damage. It does not, because a trained classifier is not a mirror. It is an optimizer, and optimizers exaggerate whatever correlation lowers their loss fastest.

How do machine learning models amplify the bias already present in training data?

A model trained to minimize average error will lean on the strongest predictive shortcut available, and a group correlation is often the cheapest shortcut in the room. The result is amplification: the output distribution is more skewed than the input distribution. In a now-canonical study, the activity “cooking” was about 33% more likely to involve women than men in the training set; the trained model pushed that association to 68% at test time (Zhao et al. 2017). Adding a corpus-level constraint at inference reduced the amplification markedly, which matters for a later point: the lever lives at inference and evaluation, not only in the data.

The data looked balanced. The model did not.

This is not a quirk of one dataset. A systematic study found amplification grows with model accuracy, capacity, and overconfidence, and with the size of the training set, and it is strongest when group membership is easier for the model to recognize than the target label itself (Hall et al. 2022). Read that condition carefully, because it inverts a comforting assumption: a more capable, better-calibrated-looking model on a larger corpus is not safer here. It is, if anything, a more efficient amplifier. Bias Amplification scales with the very things teams optimize for.

So balancing a Class Imbalance in the training data treats a symptom while leaving the mechanism intact. The model still finds the correlation, and still sharpens it.

The Impossibility Theorem: Why a Clean Dataset Still Isn’t Enough

Suppose, against everything above, you somehow obtained a perfectly balanced, perfectly measured dataset. You would hit a second wall, and this one is made of arithmetic, not data hygiene. It concerns the model’s outputs, not its inputs, which is why it is a genuinely distinct argument from the one we just made.

The Impossibility Theorem in fair classification says you cannot satisfy several intuitive fairness criteria at the same time once base rates differ between groups. When the underlying prevalence of the outcome is not identical across groups, a risk score cannot be calibrated and equalize error rates across those groups simultaneously — except in two trivial cases: perfect prediction, or identical base rates (Kleinberg et al.). Neither case describes any real high-stakes problem.

An independent proof reached the same wall from another direction. Predictive parity and error-rate balance cannot both hold when prevalence differs across groups, which means a model certified “free of predictive bias” can still produce measurable disparate impact (Chouldechova 2017). Two different formalizations, same conclusion: the conflict is built into what the fairness definitions demand of each other.

Not a bug. A definition.

This is the part that gets overstated, so be precise about its scope. The theorem does not say fairness is hopeless or unachievable. It says these specific group-fairness metrics are mutually exclusive under unequal base rates — a trade-off forced by definitions, not a verdict on the whole enterprise. In practice you can still choose which property matters most for the decision in front of you and manage the rest deliberately (FAccT 2023). What you cannot do is satisfy all of them and call the question closed.

What This Predicts for Your Pipeline

Treating these two walls as mechanisms rather than slogans turns them into predictions you can check against your own system.

• If you remove a protected attribute but leave its correlated features in place, expect the model to reconstruct the signal and the disparity to persist, now harder to see.
• If you scale up to a larger, higher-capacity, more accurate model, expect amplification to grow, not shrink — especially where group membership is easier to learn than the target label.
• If you optimize hard for one fairness metric under unequal base rates, expect another fairness metric to quietly degrade, because the theorem guarantees the trade.
• If you balance classes on the training split but let group signal cross your Cross Validation folds, expect your fairness numbers to look better than they are — that flattering gap is a form of Data Leakage.

The practical posture that follows is measurement, not eradication. You audit the model’s outputs by group, pick the fairness definition the decision actually requires, and document the residual disparity you are accepting. Open-source tooling makes this tractable: Aequitas, maintained by the Center for Data Science and Public Policy at the University of Chicago, reports group-level fairness metrics, and its v1.0.0 release (“Aequitas Flow,” 2024) added mitigation steps beyond pure auditing (Aequitas Docs). The goal it serves is measurement and choice, not elimination.

Rule of thumb: Judge disparity on the model’s outputs, not on the dataset — a balanced table tells you almost nothing about what the classifier will do with it.

When it breaks: The approach fails the moment you treat a single fairness metric as “solved.” Because calibration and error-rate balance cannot co-hold under unequal base rates, optimizing one silently worsens another, so a model that passes your chosen test can still inflict the exact disparity you were trying to prevent.

The Data Says

Bias is not a contaminant with a removal step; it is a structural property that enters through several channels, hides in correlated features, and gets amplified by the optimizer. A perfectly balanced dataset would still hit the impossibility theorem, where fairness metrics conflict by definition once base rates differ. The supportable claim is not that mitigation is pointless. It is that residual bias is unavoidable, which makes measuring and choosing your trade-off the real work.