Diffusion Transformer

A Diffusion Transformer, or DiT, is a diffusion model that uses a Transformer as its denoising backbone instead of a U-Net, operating on sequences of latent image patches.

What It Is

Classical diffusion models — Stable Diffusion 1.x and 2.x, DALL-E 2 — used a U-Net as the neural network that predicts noise at each denoising step. U-Nets were borrowed from medical image segmentation: convolutional layers with skip connections between an encoder and decoder. They worked, but they carried baked-in assumptions about locality and image structure that limit how cleanly they scale. Add more compute to a U-Net and quality gains start to flatten. The Diffusion Transformer replaces that backbone with a design whose quality climbs predictably when you feed it more parameters or more training.

The idea is simple in outline. According to Peebles & Xie (DiT), the compressed latent image produced by the VAE is cut into small patches — the same patching trick that Vision Transformers use for classification. Each patch becomes a token. A stack of standard Transformer blocks processes the sequence. At the end, the sequence is un-patchified back into a predicted-noise tensor that the scheduler uses to step the image toward a cleaner version.

The tricky part is conditioning. Diffusion models need to know two things at every step: which timestep they are denoising, and what the user asked for (a class label or a text prompt). DiT injects this information through adaptive layer normalization — specifically adaLN-Zero blocks that learn to modulate the activations based on the conditioning signal. This keeps the Transformer core unchanged while letting the model react to timestep and prompt.

The payoff is cleaner scaling. In the original paper, FID — the standard image-quality metric where lower is better — decreases monotonically as you add depth, width, or more patches. There is no sharp plateau. According to Peebles & Xie (DiT), DiT-XL/2 reached FID 2.27 on class-conditional ImageNet at 256×256, setting a new state of the art at the time.

Modern frontier models took this further. According to Esser et al. (SD3), MMDiT — the multimodal DiT variant in Stable Diffusion 3 — keeps separate weights for text tokens and image tokens, then joins them inside the attention step so the two modalities can talk. According to Black Forest Labs, FLUX.1 combines Double-Stream (multimodal) and Single-Stream (parallel) DiT blocks and is trained as a rectified-flow transformer at large parameter scale.

How It’s Used in Practice

Most people meet a DiT without realizing it. When you generate images with Stable Diffusion 3, 3.5, or FLUX.1 — either through a hosted UI like Replicate or Together, or by running the weights locally in ComfyUI or Diffusers — the network doing the actual denoising is a Diffusion Transformer, not a U-Net. The VAE still encodes and decodes, the scheduler still manages the noise timeline, and the text encoder still turns your prompt into embeddings. Only the denoiser has changed.

For practitioners fine-tuning or adapting these models, the practical impact shows up in a few places. LoRA adapters target attention and MLP layers inside the Transformer blocks rather than convolutional feature maps. Sequence length matters the way it does in language models: longer patch sequences mean higher resolution but quadratic attention cost. And because DiT scales cleanly, progressively larger community checkpoints tend to deliver real quality gains rather than diminishing returns.

Pro Tip: If you are choosing a backbone for a new image or video pipeline, start with a DiT-family model. The ecosystem around Stable Diffusion 3.5 and FLUX.1 — LoRAs, ControlNets, quantized runtimes — is now richer than what exists for legacy U-Net checkpoints, and the scaling properties mean your quality ceiling is higher.

When to Use / When Not

Common Misconception

Myth: A Diffusion Transformer is just a regular Transformer trained on images. Reality: A DiT is still a diffusion model — the training objective is noise prediction across a noise schedule, not next-token prediction. The Transformer only replaces the U-Net as the denoising network. The VAE, scheduler, and text encoder are unchanged.

One Sentence to Remember

A Diffusion Transformer is the swap that ended the U-Net era: it keeps the diffusion recipe intact but gives you a backbone whose quality climbs cleanly with compute, which is why every frontier image model shipped since Stable Diffusion 3 uses some variant of it.

FAQ

Q: Is Stable Diffusion 3 a diffusion transformer? A: Yes. Stable Diffusion 3 and 3.5 use MMDiT, a multimodal variant of DiT that keeps separate weights for text and image tokens and joins them inside the attention step of each block.

Q: What is the difference between a DiT and a Vision Transformer? A: A Vision Transformer classifies images end-to-end. A Diffusion Transformer uses the same patch-and-tokenize idea but outputs predicted noise at a given timestep, inside a diffusion denoising loop.

Q: Did DiT replace U-Nets entirely? A: For frontier text-to-image, essentially yes. U-Nets still power many open-source Stable Diffusion 1.5 and SDXL checkpoints and their LoRA ecosystems, but new frontier models ship DiT-family backbones.

Sources

• Peebles & Xie (DiT): Scalable Diffusion Models with Transformers - Original DiT paper, ICCV 2023.
• Esser et al. (SD3): Scaling Rectified Flow Transformers for High-Resolution Image Synthesis - Introduces MMDiT, the multimodal DiT used in Stable Diffusion 3.