The Fidelity-Privacy Tradeoff: Technical Limits of Anonymity Claims in Synthetic Data

Picture a dataset where every row is fabricated. No real customer, no real patient, no record copied from the source. Marketing decks call this anonymous by construction. Then someone with nothing but query access to the generator works out, with high confidence, whether your record sat in the original training set, and the fabricated rows are how they did it.

That gap between the promise and the attack is the whole subject. Synthetic Data Ethics turns on a single uncomfortable fact: a generator faithful enough to be useful is faithful enough to leak. The interesting question is not whether synthetic data can be private. It is what you have to give up to make it so.

Why a Dataset of Strangers Can Still Name You

Synthetic Data Generation starts by fitting a model to the joint distribution of the real records, then sampling new rows from it. The selling point is that no output is a literal copy of any input. The flaw hides inside the word “distribution,” because a distribution has to remember its shape, including its rarest corners.

Can synthetic data ever be truly anonymous?

The honest answer is conditional, and the condition is not the word “synthetic.” It is the attack model you defend against and the privacy budget you spend.

Consider a Membership Inference Attack: an adversary with only black-box access to a model tries to decide whether a specific record was part of its training set. This is not theoretical. Black-box access alone was enough to make this work against commercial machine-learning-as-a-service offerings from Google and Amazon, as Shokri et al. first demonstrated. A generator is a model too, and the data it emits is a downstream channel for the same signal.

The first quantitative evaluation of this question on synthetic data found something blunt: state-of-the-art generative models either failed to stop inference attacks or lost the data utility that justified using them in the first place, with no free lunch over older anonymisation methods, per Stadler et al. The protection and the usefulness pulled in opposite directions.

So a synthetic dataset is not anonymous because a model produced it. It is anonymous only when an attack against it measurably fails. Not anonymity by construction. Anonymity by configuration.

What the old anonymisation playbook got wrong

Before generative models, the standard defense was to scrub identifiers and group people together. K Anonymity formalised this: each record must be indistinguishable from at least k−1 others on its quasi-identifiers, attributes like ZIP code, birth date, and sex. The weakness is that quasi-identifiers combine, and combinations are sharp.

How sharp became clear when researchers showed that 99.98% of Americans could be correctly re-identified from just 15 demographic attributes, per Rocher et al. That figure describes traditional anonymised and sampled microdata, not synthetic data directly, but it explains the gravity the whole field is fighting. Re-Identification succeeds because real populations are full of near-unique people. A faithful generator inherits exactly that structure, which is why it can carry the same risk forward into rows that never belonged to anyone.

Turning the Privacy Dial Down Has a Price

Differential Privacy is the one privacy framework that comes with a formal, provable guarantee rather than a hopeful audit. It works by injecting calibrated noise during training, either into the gradients or into the signal a discriminator passes back, producing variants such as DP-CTGAN and PATE-GAN. The amount of noise is governed by a single number, the privacy budget ε (epsilon). Lower ε means more noise, a stronger guarantee, and less usable data.

What are the technical limits of differential privacy in synthetic data generation?

The limit is not a bug to be patched. It is the structure of the guarantee. Noise is what buys privacy, and noise is what destroys signal, so ε is a dial with utility on one end and protection on the other.

You can see the dial move in practice. In one applied evaluation, PATE-CTGAN came out ahead at a looser budget of ε ≥ 3.0, while DP-CTGAN held up better at a tight budget of ε ≤ 1.0, per Acharya et al. Treat those values as one experimental result, not an industry standard. The right ε depends on the dataset, the Benchmark Datasets you measure downstream accuracy on, and the attack you fear.

Two further limits matter. First, the plain conditional GAN baseline, CTGAN, carries no differential-privacy guarantee at all. It is built on Generative Adversarial Network training to model tabular data faithfully, and that faithfulness is precisely the property that lets an unprotected generator leak. Privacy is an added mechanism, never a default. Second, the guarantee is weakest exactly where it is needed most. The harder a record is to hide, the more noise it takes to hide it, so Memorization of outliers is the failure that no comfortable ε setting resolves.

The picture is also not fully settled. Later work has pushed back on the strongest no-free-lunch claims, arguing the trade-off softens under some configurations. The effect is reproducible across studies. Its universality is still contested, which is the honest state of the science rather than a footnote.

What the Tradeoff Predicts

Once you see privacy and fidelity as two ends of one dial, the behavior of a synthetic-data pipeline stops being mysterious and starts being predictable.

• If you raise fidelity until the synthetic rows match the real distribution on its rare combinations, expect membership inference to get easier, not harder.
• If you tighten ε toward a strong privacy guarantee, expect downstream model accuracy to drop, and expect it to drop fastest on minority subgroups where data was already thin.
• If your generator has no differential-privacy mechanism and no post-hoc attack test, expect “anonymous” to be an assumption you are making, not a property you have measured.

Rule of thumb: Treat a synthetic dataset as anonymous only after you have attacked it and the attack failed, never because a model generated it.

When it breaks: The trade-off collapses on outliers. A patient with a rare diagnosis or a customer with an unusual attribute combination sits nearly alone in the distribution, so any generator faithful enough to be useful reproduces that signal, and no ε setting hides them without degrading the data for everyone else.

Anonymous Is a Verdict, Not a Property

The most overlooked part of this is that the law already encodes the same logic the attacks measure. GDPR Recital 26 holds that data is anonymous only when re-identification is not possible by “all the means reasonably likely to be used,” judged by cost, time, and available technology, per GDPR Recital 26. That is an attacker-capability test. It is the exact variable a membership inference attack quantifies, which means the technical question and the legal question are the same question wearing different vocabulary.

The regulatory clock is also not what many slide decks claim. Under the EU AI Act, core high-risk obligations under Annex III were postponed from 2 August 2026 to 2 December 2027 by the May 2026 amendments, per Inside Privacy. That relief is narrow: GDPR continues to apply to any personal data throughout, so a delayed AI Act deadline buys no privacy holiday. If your synthetic data can be re-identified by reasonably available means, it was never anonymous under the standard that actually governs it.

The Data Says

Synthetic data shifts the privacy question; it does not answer it. Whether a generated dataset is anonymous depends on the attack you test against and the ε you are willing to spend, and the first rigorous evaluations found no configuration that delivered strong privacy and full utility at once. “Anonymous by definition” is a marketing claim the mathematics does not support.