Full search page →

Adversarial Diffusion Distillation: a teacher and a discriminator (ECCV 2024)

Optional · paper guide 5 of 7 for DLCV 12B · One-Step Generation. Citation numbers refer to the reference list at the bottom of this page.

Navigate: this page and this track: Navigate: this page and this track

On this page

Supplementary notes · 2

  1. Instantaneous velocity versus interval velocity
  2. Three meanings of teacher-free

Paper guides · 7

  1. Progressive Distillation: two teacher moves in one student move (ICLR 2022)
  2. Consistency Models: the same endpoint along a path (ICML 2023)
  3. InstaFlow: reflow, then endpoint distillation (ICLR 2024)
  4. Latent Consistency Models: endpoints in a guided latent flow (2023)
  5. Adversarial Diffusion Distillation: a teacher and a discriminator (ECCV 2024)
  6. DMD: matching a teacher-defined distribution (CVPR 2024)
  7. DMD2: tracking the student and using real data (NeurIPS 2024)

Deep Learning for Computer Vision

  1. 02 · Deep Image Classification
View the whole course tree →

Challenge

A student trained only to imitate diffusion predictions may fail to generate sharp, realistic images at very small step counts. An adversarial generator can provide strong appearance supervision, but training a large image generator from scratch is difficult. ADD combines pretrained diffusion initialization, a frozen teacher and adversarial training [14].

The design separates two signals: a discriminator evaluates realistic appearance, while a teacher-derived denoising target supplies diffusion-prior guidance. Their interaction is empirical. Neither component alone certifies prompt accuracy, and a visually sharp output is not evidence that all modes of the teacher’s distribution remain covered.

Method

The student receives both an adversarial signal and a diffusion-teacher distillation signal; training branches should not be counted as extra inference sampling steps.
Original paper figure 2 — The student receives both an adversarial signal and a diffusion-teacher distillation signal; training branches should not be counted as extra inference sampling steps.

Figure source: [14], arXiv:2311.17042v1, PDF page 2. Original PDF.

Student inputs. Real images are noised using the student’s schedule, and a timestep is drawn from a small selected set. The largest timestep has zero terminal signal so that training includes the pure-noise boundary used at generation. Intermediate timesteps retain some source-image information, enabling few-step refinement rather than limiting the model to one unconditional noise input.

Pretrained initialization. The student starts from pretrained denoiser weights. This provides a learned image prior and avoids asking the adversarial objective to discover the whole generation task from random parameters. The architecture remains a denoiser-style network with time and text conditioning. A one-step call is one trained operating point in that parameterization.

Adversarial path. Generated and real images pass through a pretrained feature network. Lightweight trainable discriminator heads operate at multiple feature levels. The heads can be conditioned on text embeddings and, when input noise retains image information, on an embedding of the corresponding source image. A hinge objective trains the discriminator; the generator is rewarded for increasing its realness scores. Regularization stabilizes the discriminator, but its score is not an independently validated image-quality probability.

Distillation path. The student’s clean prediction is noised again at a teacher noise level and passed through the frozen teacher. A stopped-gradient teacher denoising prediction becomes a reconstruction target. Noising is essential because the teacher expects a noisy input, not an arbitrary clean student output. A weighted discrepancy between the student prediction and this target supplies the distillation term. In the latent-diffusion setting, the paper finds a pixel-space distillation comparison useful for stable gradients; this involves the decoder during training.

Combined objective. The generator minimizes LG=Ladv+λLdistill\mathcal L_G=\mathcal L_{\mathrm{adv}}+\lambda\mathcal L_{\mathrm{distill}}. Here λ\lambda balances two distinct supervision sources. The teacher is fixed, the discriminator heads are trained, and the student receives gradients from both paths. Stop-gradient handling prevents treating the teacher target as a trainable collaborator. Exponential and score-distillation-inspired time weights emphasize different teacher noise levels, changing the balance between diversity and alignment.

Inference. The deployed student does not run the teacher or discriminator. It predicts from noise, with optional refinement at the trained noise levels, and the latent decoder produces pixels when applicable. The training feature network therefore affects learning cost but is not automatically part of deployment latency.

Results

The paper’s ablation configuration uses a smaller pretrained teacher/student and evaluates one student step on 5,000 COCO 2017 samples. Its default gives FID 20.6 and CLIP score 0.319. This is an ablation setting, not a universally applicable headline result for every SDXL-Turbo checkpoint.

For the larger ADD-XL model, the one-step user preference study does not beat its SDXL teacher in both categories. The four-step study reports preference over the compared teacher configuration. Those are separate budgets and experimental conditions. Replacing “four steps” with “one step” in the conclusion would overstate the paper’s result. User preference concerns the sampled images and prompts in the study; it does not directly measure rare-mode coverage.

Ablations

Table 1 makes initialization’s role clear: random initialization gives FID 293.6 and CLIP score 0.065, compared with 20.6 and 0.319 for pretrained initialization. Distillation alone gives 315.6 and 0.076; adversarial loss alone gives 20.8 and 0.315; their exponentially weighted combination gives 20.6 and 0.319. The teacher contribution therefore needs to be interpreted in interaction with the other training choices, rather than assumed to dominate the score.

Using neither image nor text discriminator conditioning gives 21.2/0.302; using both gives 20.6/0.319. The feature-network study also shows that a larger feature encoder does not necessarily improve the metric. Teacher-step counts of one, two and four give FIDs 20.6, 20.8 and 20.3 in that setup, supporting a cheap teacher target for the selected training configuration without proving that all longer teacher trajectories are unnecessary.

Conclusion

A large pretrained diffusion student can learn a very short generation procedure using both adversarial appearance supervision and a teacher-derived denoising signal. Pretraining and conditioning are important parts of the system.

Problems

Adversarial optimization can favor attractive but repetitive detail. The method needs careful loss weighting, data and discriminator design, and quality depends on the inference budget. Later LADD changes training representations to reduce image-feature and decoding overhead [15]. Speed comparisons must retain that distinction: cheaper training representations do not eliminate the final image decoder or guarantee the same objective at every scale.

References

  1. Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, B. Poole, “Score-Based Generative Modeling through Stochastic Differential Equations,” ICLR, 2021. paper
  2. Y. Lipman, M. Havasi, P. Holderrieth, et al., “Flow Matching Guide and Code,” arXiv tutorial, 2024. paper
  3. T. Karras, M. Aittala, T. Aila, S. Laine, “Elucidating the Design Space of Diffusion-Based Generative Models,” NeurIPS, 2022. source
  4. X. Fan, Z. Wu, H. Wu, L. Cao, “A Survey on Pre-Trained Diffusion Model Distillations,” arXiv preprint, version 2, 2025. paper
  5. J. Song, C. Meng, S. Ermon, “Denoising Diffusion Implicit Models,” ICLR, 2021. paper
  6. C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li, J. Zhu, “DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps,” NeurIPS, 2022. paper
  7. T. Salimans, J. Ho, “Progressive Distillation for Fast Sampling of Diffusion Models,” ICLR, 2022. paper
  8. Y. Song, P. Dhariwal, M. Chen, I. Sutskever, “Consistency Models,” ICML, 2023. paper
  9. Y. Song, P. Dhariwal, “Improved Techniques for Training Consistency Models,” ICLR, 2024. paper
  10. S. Luo, Y. Tan, L. Huang, J. Li, H. Zhao, “Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference,” arXiv preprint, 2023. paper
  11. R. Rombach, A. Blattmann, D. Lorenz, P. Esser, B. Ommer, “High-Resolution Image Synthesis with Latent Diffusion Models,” CVPR, 2022. source
  12. X. Liu, C. Gong, Q. Liu, “Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow,” ICLR, 2023. paper
  13. X. Liu, X. Zhang, J. Ma, J. Peng, Q. Liu, “InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation,” ICLR, 2024. paper
  14. A. Sauer, D. Lorenz, A. Blattmann, R. Rombach, “Adversarial Diffusion Distillation,” ECCV, 2024. paper
  15. A. Sauer, F. Boesel, T. Dockhorn, A. Blattmann, P. Esser, R. Rombach, “Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion Distillation,” SIGGRAPH Asia Conference Papers, 2024. paper · published version
  16. T. Yin, M. Gharbi, R. Zhang, E. Shechtman, F. Durand, W. T. Freeman, T. Park, “One-step Diffusion with Distribution Matching Distillation,” CVPR, 2024. paper
  17. T. H. Nguyen, A. Tran, “SwiftBrush: One-Step Text-to-Image Diffusion Model with Variational Score Distillation,” CVPR, 2024. paper
  18. M. Zhou, H. Zheng, Z. Wang, M. Yin, H. Huang, “Score identity Distillation: Exponentially Fast Distillation of Pretrained Diffusion Models for One-Step Generation,” ICML, 2024. paper
  19. T. Yin, M. Gharbi, T. Park, R. Zhang, E. Shechtman, F. Durand, W. T. Freeman, “Improved Distribution Matching Distillation for Fast Image Synthesis,” NeurIPS, 2024. paper
  20. K. Frans, D. Hafner, S. Levine, P. Abbeel, “One Step Diffusion via Shortcut Models,” ICLR, 2025. paper
  21. Z. Geng, M. Deng, X. Bai, J. Z. Kolter, K. He, “Mean Flows for One-step Generative Modeling,” NeurIPS, 2025. paper
  22. M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, S. Hochreiter, “GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium,” NeurIPS, 2017. source
  23. A. Radford, J. W. Kim, C. Hallacy, et al., “Learning Transferable Visual Models From Natural Language Supervision,” ICML, 2021. paper
  24. D. Ghosh, H. Hajishirzi, L. Schmidt, “GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment,” NeurIPS Datasets and Benchmarks, 2023. paper
  25. A. Krizhevsky, V. Nair and G. Hinton, “CIFAR-10 and CIFAR-100 datasets,” official dataset page; companion technical report: A. Krizhevsky, “Learning Multiple Layers of Features from Tiny Images,” 2009. dataset and report
  26. J. Deng, W. Dong, R. Socher, L. Li, Kai Li, Li Fei-Fei, “ImageNet: A large-scale hierarchical image database,” CVPR, 2009. doi
  27. T. Lin, M. Maire, S. Belongie, et al., “Microsoft COCO: Common Objects in Context,” ECCV, 2014. paper
  28. L. Yang, Z. Zhang, Y. Song, et al., “Diffusion Models: A Comprehensive Survey of Methods and Applications,” ACM Computing Surveys, 2023. paper
  29. Z. Yu, J. Yuan, H. Yang, et al., “DMAD: Distribution Matching as Adversarial Distillation for Fast Visual Generation,” arXiv preprint, 2026. paper
  30. R. Zhang, P. Isola, A. A. Efros, E. Shechtman and O. Wang, “The Unreasonable Effectiveness of Deep Features as a Perceptual Metric,” in Proc. IEEE/CVF CVPR, 2018, pp. 586–595. doi · arXiv
  31. C. Schuhmann, R. Beaumont, R. Vencu, et al., “LAION-5B: An open large-scale dataset for training next generation image-text models,” NeurIPS Datasets and Benchmarks, 2022. paper