From GANs to Diffusion Models: Prerequisites and Technical Limits of AI Avatar Generation

Ask an engineer what changed between a 2021 avatar demo and a 2026 deployment, and most will say “the model got bigger.” That’s not quite what happened. The 2D face-synthesis method was replaced wholesale — adversarial training swapped for diffusion — and the 3D reconstruction method underneath was replaced too, on its own separate timeline. Two independent architecture shifts, stacked on top of each other, each fixing a different failure mode and leaving another untouched.

Two Generation Stacks Hiding Inside Every Avatar

Strip away the product UI and AI Avatar Generation turns out to be two separate pattern-generation problems wearing one interface. One produces the face: skin texture, lip movement, micro-expression, frame by frame. The other produces the geometry: how that face sits in 3D space when the camera — real or virtual — moves around it. Most production tools today only need the first; full 3D avatars need both, and they evolved on completely different schedules.

What background knowledge do I need before working with AI avatar generation systems?

The face-synthesis side starts with the Generative Adversarial Network: two networks, a generator and a discriminator, trained against each other until the generator’s fakes pass as real, an architecture introduced in 2014 by Goodfellow et al. (arXiv). For nearly a decade it was the default engine for Talking Head Synthesis, animating a still photo into a speaking face synced to audio. The catch is structural: adversarial training is notoriously unstable. Push the generator too far ahead of the discriminator and the network stops exploring the space of possible faces and collapses onto the same few safe outputs regardless of input — Mode Collapse.

By the early 2020s, Diffusion Models displaced GANs across most of generative vision, and talking-head synthesis followed. Diffusion systems learn to reverse a noise process step by step instead of racing an adversary, which trades GAN-style instability for a different cost — many denoising passes per frame instead of one forward pass through a generator. Production vendors made that trade anyway. The first diffusion-based talking-head model, Diffused Heads, arrived in 2023 (Diffused Heads paper), and diffusion has since overtaken GANs on image fidelity and lip-sync accuracy. Mode collapse, the GAN-era failure mode, comes from adversarial training, so a diffusion model that races no discriminator isn’t prone to it in the first place.

The geometry side has its own, unrelated history. NeRF, introduced in 2020 (Mildenhall et al., arXiv), represents a 3D scene as a neural function queried at any viewpoint — accurate, but slow enough to train that real-time avatar rendering stayed impractical for years. Gaussian Splatting replaced it for production use: instead of querying a network per pixel, the scene becomes millions of explicit, optimizable 3D Gaussians that a GPU rasterizer renders in real time, a technique published in 2023 (Kerbl et al., arXiv). By 2026, Gaussian Splatting had become the de facto production standard — OpenUSD and the Khronos glTF format both added native support for it (The Future 3D) — while NeRF settled into a research and benchmarking role rather than disappearing outright.

Both of those are reconstruction methods: they need a 3D scene, or many 2D views of one, to learn from. Text-to-3D generation has no such input — it has to invent geometry from a text prompt alone. The technique that makes this possible, Score Distillation Sampling, doesn’t train a new 3D model at all. It repurposes a pretrained 2D diffusion model as a critic, iteratively nudging a 3D representation until renders from every angle satisfy that critic, a method DreamFusion introduced to distill a 2D diffusion model’s priors into a NeRF scene without a single 3D training example (DreamFusion paper). Not a new generative architecture. A way of borrowing one.

There’s a fifth prerequisite that has nothing to do with model architecture: knowing where Deepfake regulation draws its lines, because every one of these pipelines can produce output that regulators now treat as a labeling problem, not just an engineering one.

Why HeyGen and Synthesia Both Abandoned GANs

Research timelines and product timelines rarely move at the same speed, and the gap between them is usually where the real signal sits. GAN-based talking-head models were a known dead end in papers well before commercial avatar platforms stopped shipping them — and then, within about six months of each other, both of the category’s leading vendors rebuilt their core models from scratch.

HeyGen’s current production model, Avatar V, is a diffusion transformer trained with flow matching rather than the older fixed denoising schedule; it conditions on the full token sequence of a reference video instead of a single reference frame, a method the company calls Sparse Reference Attention, released April 8, 2026 (HeyGen Research). Synthesia made the same architectural move from a different angle: Express-2, the engine behind Synthesia 3.0 (launched October 1, 2025), is also a Diffusion Transformer model, built specifically to handle full-body gesture rather than head-and-shoulders framing alone (Synthesia Blog).

Both rebuilt the generative core. Neither company frames the change as a marketing refresh, and as of mid-2026 they’re the two dominant commercial avatar platforms — both now running diffusion transformers instead of the GAN pipelines that defined the category five years earlier. What neither launch announcement mentions is what’s still broken underneath the new architecture.

Where the Stack Still Breaks

Architecture upgrades fix the failure mode they were built to fix. They rarely fix the others. A handful of limitations have survived every generation of this stack so far, and one more — not technical at all — just became a hard deadline.

What are the main technical limitations of AI avatar generation in 2026?

The most persistent limitation is identity drift: a face that subtly stops matching itself across a long take, more noticeable the longer the clip runs. HeyGen reports zero observed identity drift on clips over four minutes with Avatar V, against visible drift on the prior Avatar IV model — a self-reported benchmark, not independently verified, and worth treating as a vendor claim rather than a settled result (HeyGen Research). Synthesia’s own announcement for Express-2 doesn’t name a failure mode at all, which is typical of vendor blog posts and is not evidence that one doesn’t exist.

The architecture trade introduced earlier hasn’t gone away — it moved. GANs were unstable to train but cheap to run; diffusion transformers are stable to train but expensive to run, since each frame still needs multiple denoising passes instead of one pass through a generator. Real-time, full-resolution avatar streaming inherits that cost directly. On the geometry side, Gaussian Splatting solved NeRF's speed problem but inherited a different one: an explicit cloud of millions of Gaussians is heavier to store and stream than a compact neural function, part of why NeRF stays useful as a research baseline even as production pipelines move past it.

The fourth limitation isn’t in the model at all — it’s in the surface around it. HeyGen’s own API is mid-migration, with several endpoints already scheduled for retirement, and new disclosure rules are stacking onto every one of these pipelines through 2026: a New York law requiring synthetic-performer disclosure took effect in June, and the EU AI Act’s Article 50 deepfake-labeling rule becomes enforceable in August, with machine-readable marking for systems already on the market extended to December (EU AI Act explainer). A model that perfectly solved identity drift would still not satisfy a labeling requirement. Deepfake disclosure is now a deployment constraint, not a footnote.

Platform & compliance notes:

• HeyGen API deprecations: v1/v2 endpoints retire October 31, 2026; the Streaming API is deprecated in favor of LiveAvatar; Interactive Avatar sunsets March 31, 2026. Action: migrate integrations before these dates.
• Disclosure deadlines stacking in 2026: New York’s synthetic-performer disclosure law took effect June 9, 2026; the EU AI Act’s Article 50 deepfake-labeling rule becomes enforceable August 2, 2026, with machine-readable marking for pre-market systems extended to December 2, 2026. Action: build disclosure labeling into any avatar deployment now.

If you’re integrating a vendor avatar API today, treat the architecture as unstable infrastructure, not a fixed contract — expect at least one underlying model swap within any given product relationship. If a platform’s marketing leads with identity stability or limitation-free claims, expect those claims to be self-reported until you test them on your own footage.

Rule of thumb: test for identity drift on your own longest expected clip length; vendor benchmarks run on their best-case footage, not yours.

When it breaks: diffusion transformer avatars still drift on long, high-motion takes under inconsistent lighting, and no current vendor — including the two covered here — publishes failure-rate data alongside its fidelity claims.

The Data Says

Two architecture shifts converged on the same platforms in the same year: GANs gave way to diffusion transformers for the face, NeRF gave way to Gaussian Splatting for the geometry, and Score Distillation Sampling lets either stack invent 3D content with no 3D training data at all. None of that touched identity drift, inference cost, or the disclosure rules now layered on top of every deployment — the math under the hood and the constraints around it are solving entirely different problems.