What Is Synthetic Data Ethics and How Generated Records Can Re-Identify Real People

Picture a dataset where not one row belongs to a real person. Every record was sampled from a model, no customer copied, no patient duplicated. A vendor stamps it anonymous and hands it to a partner. Months later, someone holding only that file works out, with high confidence, that a specific real person sat in the cohort it learned from.

The unsettling part is that the fabricated rows are how they did it. Synthetic Data Ethics is the discipline that takes this gap seriously: the distance between what a generated dataset promises and what an attacker can actually recover from it.

The Field That Begins Where “Just Fake Data” Ends

The phrase sounds like a contradiction. If the records are invented, who is left to wrong? The answer is everyone whose real data taught the model what to invent.

What is synthetic data ethics?

Synthetic data ethics is the study of the obligations, privacy, fairness, and consent, that survive the act of generating artificial records modeled on real ones. It is not a property of a file; it is a property of the pipeline that produced it. Synthetic Data Generation is the technique: fit a model to the statistical shape of a population, then sample new rows from that shape. Ethics asks the second question the technique skips, which is what is owed to the people whose data defined the shape.

Not fake people. Compressed real ones.

The discipline organizes around three failure modes that the word “synthetic” tends to paper over: a generated record can point back to an identifiable person, it can carry forward and sharpen the discrimination baked into its source, and it can quietly relicense data for uses nobody agreed to. A record being non-copied is not the same as a person being unexposed, and most of the field exists to keep those two ideas from being confused.

How does AI-generated synthetic data leak information about real individuals?

A generator is, mechanically, a lossy compressor of a population. To reproduce the common patterns it can average freely, but the rare records carry the most information, the highest surprise, so a model faithful enough to be useful has to spend representational capacity preserving them. The outliers are simultaneously the hardest rows to fake convincingly and the easiest people to find. A model faithful enough to be useful is faithful enough to leak.

A Membership Inference Attack formalizes the threat. Given only black-box access to a model and a candidate record, an adversary decides whether that record was in the training set, and black-box access alone was enough to make this work against deployed commercial models, as Shokri et al. first showed. A generator is a model too, and the rows it emits are one more channel carrying the same signal.

Re-Identification explains why the signal is so potent. Sweeney’s classic study found that roughly 87% of the U.S. population was uniquely identifiable from just ZIP code, sex, and date of birth, per Sweeney. That figure comes from 1990 census data, and later replications on newer data put it closer to 63%, but the direction holds either way: real populations are full of near-unique people, and a faithful generator inherits that structure rather than erasing it.

The first quantitative evaluation of the question reached a blunt verdict. Across state-of-the-art generative models, synthetic data either failed to stop inference attacks or lost the utility that justified using it, with no better privacy-utility tradeoff than the older anonymisation methods it was meant to replace, per Stadler et al. The protection and the usefulness pulled in opposite directions.

The same logic reaches past tabular vendors. Knowledge Distillation, where a student model trains on a teacher’s generated outputs, produces synthetic data too, and inherits the same unanswered question about how much of the teacher’s training set rides along inside it.

Three Harms a Dataset of Strangers Can Still Cause

Privacy leakage is the sharpest risk, and the only one with mature attack tooling, but it is not the only one. Two more sit beside it, and they are harder to measure precisely because they are not about a single identifiable person.

What are the main ethical risks of synthetic data?

The first risk is re-identification: a generated record that points back to a real individual. This is the mechanism above, and it is the one a membership inference attack can quantify against a specific model.

The second is Bias Amplification. A generator learns the distribution it is given, including its skew. Train on data where a minority subgroup is thin and noisy, and the model holds the least evidence exactly where the stakes run highest; sampling from it can thin that group further or harden a correlation the original data only hinted at. A fabricated dataset can launder a biased one into something that looks neutral because it looks new. This is a framing concept rather than a statute, and the magnitude depends on the generator and the source data, but the direction of the effect is consistent enough to plan around.

The third is Consent Laundering. People in the original dataset agreed, if they agreed at all, to one use under one set of assumptions about who would see them. Calling the derived data “synthetic, therefore not personal” can relicense it for a second life nobody described to them. Whether that move is legitimate turns on a legal test, not a marketing claim.

That legal test is more aligned with the technical one than most decks admit. GDPR treats data as anonymous only when re-identification is not possible by the means reasonably likely to be used, judged by cost, time, and available technology. The UK’s regulator operationalizes this with a “motivated intruder” test and requires the assessment to be revisited over time, per ICO; identifiability is a spectrum, not a stamp. The EU AI Act, in force since 2024, references synthetic data as a privacy-preserving option within its data-governance provisions but does not set the re-identification bar, which remains GDPR’s domain. A provisionally agreed Digital Omnibus would postpone the Act’s high-risk obligations into late 2027, per Gibson Dunn, yet a delayed AI deadline changes nothing about the privacy standard that already governs the underlying personal data.

The Concepts That Separate “Private” From “Private-Looking”

Before you can judge any “privacy-safe” claim, you need the handful of ideas the claim is implicitly making, or quietly skipping. Each one is a place where a generator either earns the label or only borrows it.

What concepts do you need to understand before evaluating synthetic data privacy?

K Anonymity is the older defense: each record must be indistinguishable from at least k−1 others on its quasi-identifiers, attributes like ZIP code, birth date, and sex, a model Sweeney formalized. Its weakness is the one re-identification exploits, since quasi-identifiers combine and combinations are sharp.

Differential Privacy is the one framework that comes with a provable, per-individual guarantee instead of a hopeful audit. Introduced by Dwork et al. in 2006, it bounds how much any single record can change a model’s output, governed by a privacy budget called epsilon; a lower budget means more injected noise, a stronger guarantee, and less usable data. The catch lives in the implementation: common differentially private synthesis tools were found to violate their own theoretical assumptions in practice, leaving exactly the outlier records exposed, the Memorization failure that no comfortable budget resolves, per Stadler et al. A “DP” label is a claim to verify, not a guarantee to trust.

CTGAN, the canonical conditional-GAN baseline for tabular data and part of the open-source Synthetic Data Vault, carries no differential-privacy guarantee by default, per SDV’s GitHub repository. Its entire design goal is fidelity, modeling continuous and categorical columns faithfully, which is precisely the property that lets an unprotected generator leak. Privacy is a mechanism you add, never a default you inherit.

The last piece is measurement. Whatever budget you choose, you only learn its cost by scoring downstream models on Benchmark Datasets and attacking the generator to see what survives. Without both numbers, “private” and “private-looking” stay indistinguishable.

Reading the Risk Before You Trust the Label

Once you see the generator as a faithful compressor governed by a privacy dial, the behavior of a synthetic-data pipeline stops being mysterious and starts being predictable.

• If a generator is tuned for high fidelity on rare combinations, expect membership inference to get easier, not harder, because faithfulness on the tail is exactly what an attacker reads.
• If you tighten the privacy budget toward a strong guarantee, expect downstream accuracy to fall first and fastest on the minority subgroups that were already thin, where bias amplification and privacy loss meet.
• If a dataset arrives labeled “anonymous” with no named attack model and no stated budget, expect it to mean nobody tested, not nobody exposed.

The practical move is to treat every privacy and fairness claim as a measurement you demand, not an adjective you accept.

Rule of thumb: Treat a synthetic dataset as anonymous only after you have attacked it and the attack failed, and as fair only after you have measured the subgroup it represents worst.

When it breaks: The whole framework collapses on outliers. A person with a rare diagnosis or an unusual combination of traits sits nearly alone in the distribution, so any generator faithful enough to be useful reproduces their signal, and no privacy budget hides them without degrading the data for everyone else.

The Data Says

Synthetic data ethics is not a verdict the word “synthetic” delivers; it is one you earn by measuring. A generator faithful enough to be useful preserves the rare individuals it learned from, so privacy depends on the attack you test and the budget you spend, and the same faithfulness that protects no one on the margins can concentrate bias and relicense consent nobody gave. The mathematics does not support “ethical by default.”