Adversarial Diffusion Distillation: a teacher and a discriminator (ECCV 2024)
Optional · paper guide 5 of 7 for DLCV 12B · One-Step Generation. Citation numbers refer to the reference list at the bottom of this page.
Navigate: this page and this track: Navigate: this page and this track
On this page
- Progressive Distillation: two teacher moves in one student move (ICLR 2022)
- Consistency Models: the same endpoint along a path (ICML 2023)
- InstaFlow: reflow, then endpoint distillation (ICLR 2024)
- Latent Consistency Models: endpoints in a guided latent flow (2023)
- Adversarial Diffusion Distillation: a teacher and a discriminator (ECCV 2024)
- DMD: matching a teacher-defined distribution (CVPR 2024)
- DMD2: tracking the student and using real data (NeurIPS 2024)
Deep Learning for Computer Vision
Challenge
A student trained only to imitate diffusion predictions may fail to generate sharp, realistic images at very small step counts. An adversarial generator can provide strong appearance supervision, but training a large image generator from scratch is difficult. ADD combines pretrained diffusion initialization, a frozen teacher and adversarial training [14].
The design separates two signals: a discriminator evaluates realistic appearance, while a teacher-derived denoising target supplies diffusion-prior guidance. Their interaction is empirical. Neither component alone certifies prompt accuracy, and a visually sharp output is not evidence that all modes of the teacher’s distribution remain covered.
Method

Figure source: [14], arXiv:2311.17042v1, PDF page 2. Original PDF.
Student inputs. Real images are noised using the student’s schedule, and a timestep is drawn from a small selected set. The largest timestep has zero terminal signal so that training includes the pure-noise boundary used at generation. Intermediate timesteps retain some source-image information, enabling few-step refinement rather than limiting the model to one unconditional noise input.
Pretrained initialization. The student starts from pretrained denoiser weights. This provides a learned image prior and avoids asking the adversarial objective to discover the whole generation task from random parameters. The architecture remains a denoiser-style network with time and text conditioning. A one-step call is one trained operating point in that parameterization.
Adversarial path. Generated and real images pass through a pretrained feature network. Lightweight trainable discriminator heads operate at multiple feature levels. The heads can be conditioned on text embeddings and, when input noise retains image information, on an embedding of the corresponding source image. A hinge objective trains the discriminator; the generator is rewarded for increasing its realness scores. Regularization stabilizes the discriminator, but its score is not an independently validated image-quality probability.
Distillation path. The student’s clean prediction is noised again at a teacher noise level and passed through the frozen teacher. A stopped-gradient teacher denoising prediction becomes a reconstruction target. Noising is essential because the teacher expects a noisy input, not an arbitrary clean student output. A weighted discrepancy between the student prediction and this target supplies the distillation term. In the latent-diffusion setting, the paper finds a pixel-space distillation comparison useful for stable gradients; this involves the decoder during training.
Combined objective. The generator minimizes . Here balances two distinct supervision sources. The teacher is fixed, the discriminator heads are trained, and the student receives gradients from both paths. Stop-gradient handling prevents treating the teacher target as a trainable collaborator. Exponential and score-distillation-inspired time weights emphasize different teacher noise levels, changing the balance between diversity and alignment.
Inference. The deployed student does not run the teacher or discriminator. It predicts from noise, with optional refinement at the trained noise levels, and the latent decoder produces pixels when applicable. The training feature network therefore affects learning cost but is not automatically part of deployment latency.
Results
The paper’s ablation configuration uses a smaller pretrained teacher/student and evaluates one student step on 5,000 COCO 2017 samples. Its default gives FID 20.6 and CLIP score 0.319. This is an ablation setting, not a universally applicable headline result for every SDXL-Turbo checkpoint.
For the larger ADD-XL model, the one-step user preference study does not beat its SDXL teacher in both categories. The four-step study reports preference over the compared teacher configuration. Those are separate budgets and experimental conditions. Replacing “four steps” with “one step” in the conclusion would overstate the paper’s result. User preference concerns the sampled images and prompts in the study; it does not directly measure rare-mode coverage.
Ablations
Table 1 makes initialization’s role clear: random initialization gives FID 293.6 and CLIP score 0.065, compared with 20.6 and 0.319 for pretrained initialization. Distillation alone gives 315.6 and 0.076; adversarial loss alone gives 20.8 and 0.315; their exponentially weighted combination gives 20.6 and 0.319. The teacher contribution therefore needs to be interpreted in interaction with the other training choices, rather than assumed to dominate the score.
Using neither image nor text discriminator conditioning gives 21.2/0.302; using both gives 20.6/0.319. The feature-network study also shows that a larger feature encoder does not necessarily improve the metric. Teacher-step counts of one, two and four give FIDs 20.6, 20.8 and 20.3 in that setup, supporting a cheap teacher target for the selected training configuration without proving that all longer teacher trajectories are unnecessary.
Conclusion
A large pretrained diffusion student can learn a very short generation procedure using both adversarial appearance supervision and a teacher-derived denoising signal. Pretraining and conditioning are important parts of the system.
Problems
Adversarial optimization can favor attractive but repetitive detail. The method needs careful loss weighting, data and discriminator design, and quality depends on the inference budget. Later LADD changes training representations to reduce image-feature and decoding overhead [15]. Speed comparisons must retain that distinction: cheaper training representations do not eliminate the final image decoder or guarantee the same objective at every scale.
References
- Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, B. Poole, “Score-Based Generative Modeling through Stochastic Differential Equations,” ICLR, 2021. paper
- Y. Lipman, M. Havasi, P. Holderrieth, et al., “Flow Matching Guide and Code,” arXiv tutorial, 2024. paper
- T. Karras, M. Aittala, T. Aila, S. Laine, “Elucidating the Design Space of Diffusion-Based Generative Models,” NeurIPS, 2022. source
- X. Fan, Z. Wu, H. Wu, L. Cao, “A Survey on Pre-Trained Diffusion Model Distillations,” arXiv preprint, version 2, 2025. paper
- J. Song, C. Meng, S. Ermon, “Denoising Diffusion Implicit Models,” ICLR, 2021. paper
- C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li, J. Zhu, “DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps,” NeurIPS, 2022. paper
- T. Salimans, J. Ho, “Progressive Distillation for Fast Sampling of Diffusion Models,” ICLR, 2022. paper
- Y. Song, P. Dhariwal, M. Chen, I. Sutskever, “Consistency Models,” ICML, 2023. paper
- Y. Song, P. Dhariwal, “Improved Techniques for Training Consistency Models,” ICLR, 2024. paper
- S. Luo, Y. Tan, L. Huang, J. Li, H. Zhao, “Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference,” arXiv preprint, 2023. paper
- R. Rombach, A. Blattmann, D. Lorenz, P. Esser, B. Ommer, “High-Resolution Image Synthesis with Latent Diffusion Models,” CVPR, 2022. source
- X. Liu, C. Gong, Q. Liu, “Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow,” ICLR, 2023. paper
- X. Liu, X. Zhang, J. Ma, J. Peng, Q. Liu, “InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation,” ICLR, 2024. paper
- A. Sauer, D. Lorenz, A. Blattmann, R. Rombach, “Adversarial Diffusion Distillation,” ECCV, 2024. paper
- A. Sauer, F. Boesel, T. Dockhorn, A. Blattmann, P. Esser, R. Rombach, “Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion Distillation,” SIGGRAPH Asia Conference Papers, 2024. paper · published version
- T. Yin, M. Gharbi, R. Zhang, E. Shechtman, F. Durand, W. T. Freeman, T. Park, “One-step Diffusion with Distribution Matching Distillation,” CVPR, 2024. paper
- T. H. Nguyen, A. Tran, “SwiftBrush: One-Step Text-to-Image Diffusion Model with Variational Score Distillation,” CVPR, 2024. paper
- M. Zhou, H. Zheng, Z. Wang, M. Yin, H. Huang, “Score identity Distillation: Exponentially Fast Distillation of Pretrained Diffusion Models for One-Step Generation,” ICML, 2024. paper
- T. Yin, M. Gharbi, T. Park, R. Zhang, E. Shechtman, F. Durand, W. T. Freeman, “Improved Distribution Matching Distillation for Fast Image Synthesis,” NeurIPS, 2024. paper
- K. Frans, D. Hafner, S. Levine, P. Abbeel, “One Step Diffusion via Shortcut Models,” ICLR, 2025. paper
- Z. Geng, M. Deng, X. Bai, J. Z. Kolter, K. He, “Mean Flows for One-step Generative Modeling,” NeurIPS, 2025. paper
- M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, S. Hochreiter, “GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium,” NeurIPS, 2017. source
- A. Radford, J. W. Kim, C. Hallacy, et al., “Learning Transferable Visual Models From Natural Language Supervision,” ICML, 2021. paper
- D. Ghosh, H. Hajishirzi, L. Schmidt, “GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment,” NeurIPS Datasets and Benchmarks, 2023. paper
- A. Krizhevsky, V. Nair and G. Hinton, “CIFAR-10 and CIFAR-100 datasets,” official dataset page; companion technical report: A. Krizhevsky, “Learning Multiple Layers of Features from Tiny Images,” 2009. dataset and report
- J. Deng, W. Dong, R. Socher, L. Li, Kai Li, Li Fei-Fei, “ImageNet: A large-scale hierarchical image database,” CVPR, 2009. doi
- T. Lin, M. Maire, S. Belongie, et al., “Microsoft COCO: Common Objects in Context,” ECCV, 2014. paper
- L. Yang, Z. Zhang, Y. Song, et al., “Diffusion Models: A Comprehensive Survey of Methods and Applications,” ACM Computing Surveys, 2023. paper
- Z. Yu, J. Yuan, H. Yang, et al., “DMAD: Distribution Matching as Adversarial Distillation for Fast Visual Generation,” arXiv preprint, 2026. paper
- R. Zhang, P. Isola, A. A. Efros, E. Shechtman and O. Wang, “The Unreasonable Effectiveness of Deep Features as a Perceptual Metric,” in Proc. IEEE/CVF CVPR, 2018, pp. 586–595. doi · arXiv
- C. Schuhmann, R. Beaumont, R. Vencu, et al., “LAION-5B: An open large-scale dataset for training next generation image-text models,” NeurIPS Datasets and Benchmarks, 2022. paper