Bias In, Bias Amplified: Does Synthetic Data Clean or Concentrate Discrimination?

Every dataset is a portrait of who got counted and who did not. When we feed that portrait into a generator and ask it to paint more faces, we are not escaping the original framing. We are scaling it. The only honest question is whether the scaling corrects the distortion or deepens it, and whether anyone is left accountable for the answer.

When We Manufacture Data, Where Does the Bias Go?

The promise arrives wrapped in good intentions. Synthetic Data Generation — using models like GANs, diffusion systems, and variational autoencoders to manufacture records that resemble real ones — lets an organization sidestep the cost, the privacy exposure, and the imbalance of real-world data. Feed it a skewed dataset, ask for a balanced one, and the discrimination appears to dissolve. The field of Synthetic Data Ethics exists because that appearance deserves scrutiny. If a model learns the world from data that already encodes who was overpoliced, underdiagnosed, or simply overlooked, then manufacturing more of that data raises a question we tend to skip past in the rush to a cleaner training set: where does the bias actually go?

The Honest Case for Manufactured Fairness

The optimistic case is not naive, and it deserves to be stated at full strength. When a group is underrepresented, you can deliberately generate more examples of it, rebalancing a training set or padding thin Benchmark Datasets until rare cases carry statistical weight. Fairness-aware generation methods do exactly this on purpose, and they work: bias-transforming approaches can reduce the representation gap rather than inherit it (Bt-GAN study). Add the privacy argument — synthetic records let researchers study sensitive patterns without exposing the individuals behind them — and synthetic data starts to look less like a shortcut and more like a moral upgrade over the messy, consent-fraught data it replaces. The case is real. It is also built on an assumption worth examining.

The Assumption Hiding Inside the Fix

The assumption is that bias is a problem of proportions — too few examples of one group — and that proportions are something you can manufacture your way out of. But discrimination in data is rarely only a counting error. Much of it is systemic, baked into which outcomes were ever recorded as “normal” and which were filtered out before a dataset existed. A generator trained on that source does not perceive the injustice; it perceives a distribution, and it reproduces the distribution faithfully. This is where Bias Amplification enters: when a group is sparse, a model holds the least information about it, so its synthetic version comes out the least accurate and the most stereotyped. The very populations the technique is meant to protect are the ones it reconstructs worst.

The Copy That Forgets Its Outliers

There is a sharper way to see what generation really does: it behaves less like a mirror than a summary, and summaries lose their outliers first. Train a model on data, generate from it, train the next model on that output, and the rare tails erode with each pass — researchers writing in Nature documented exactly this collapse, where minority features are the first to vanish when models learn from recursively generated data (Nature). The privacy fixes can make it worse. Applying differential privacy to synthetic data tends to distort underrepresented groups the most, because protecting an unusual record means blurring it (2025 literature review). Picture the old composite portraits that averaged many faces into one “typical” face: smooth, plausible, and a likeness of no one who actually exists. Manufactured data drifts toward that average, and the people at the edges quietly fall out of the frame.

The Question Underneath the Question

So does synthetic data clean discrimination or concentrate it? The honest answer is that it depends on the method, the intent, and the auditing — which is precisely why the clean-versus-concentrate framing misleads. It treats the outcome as a property of the technology, when the outcome is a property of the choices made around it.

Thesis: Synthetic data does not choose between cleaning and concentrating discrimination; it chooses who can no longer be held responsible for the discrimination that survives.

Manufacturing a dataset inserts a layer between the people who might be harmed and the institution making the decision. Researchers at FAccT have shown how synthetic pipelines can enable diversity-washing and circumvent consent, decoupling the data from the people it represents (FAccT 2024). Some critics call the sharper version of this Consent Laundering: real information seeds a generator, the output is declared new and consent-free, and the original subjects lose any say in what their data becomes. Whether or not that phrase ever hardens into a legal category, the mechanism is already here. The bias may shrink or grow. The accountability almost always thins.

Whose Hands Are on the Data?

That thinning of accountability is the part that should keep us up at night, because the privacy promise turns out to be just as conditional as the fairness one. Older anonymization models like K Anonymity tried to hide a person inside a crowd of identical records, and GDPR principles still locate genuine anonymization at the point where an individual can no longer be singled out, linked, or inferred. Synthetic data is frequently sold as clearing that bar by default. It does not. A Membership Inference Attack can reveal whether a specific person’s record shaped the generator, and Re-Identification stays possible whenever the synthetic distribution hugs the real one too closely (2026 arXiv study). “Fake” is not the same as “anonymous.” So when a manufactured dataset produces a discriminatory outcome, who answers for it — the company that bought the generator, the vendor that trained it, or the people whose records seeded a distribution they never agreed to join?

Where This Argument Could Break

This is the point where I have to argue against myself. If provenance standards mature, if fairness-aware generation becomes the default rather than the exception, and if auditing tools can trace which real population each synthetic record stands in for, then manufactured data could end up more accountable than the opaque, scraped datasets it replaces. The EU AI Act points that way in principle: its data-governance rules hold that the data used to train and test a model should be representative, and that providers should work to detect and mitigate bias (EU AI Act). The timeline tells its own story. The bloc’s May 2026 Digital Omnibus pushed the high-risk obligations to December 2027 (EU Council), which either gives the safeguards time to mature or gives the accountability gap time to harden. If the safeguards win that race, my thesis weakens considerably.

The Question That Remains

Synthetic data will not be the thing that makes AI fair, nor the thing that makes it unjust. It is a layer of abstraction, and abstractions hold power precisely because they let us stop looking at what they replace. The real question is not whether the data is real. It is whether, once we can manufacture any population we like, we will still feel responsible for the people the manufactured version leaves out.

Ethically, Alan.