DMD: matching a teacher-defined distribution (CVPR 2024)
Optional · paper guide 6 of 7 for DLCV 12B · One-Step Generation. Citation numbers refer to the reference list at the bottom of this page.
Navigate: this page and this track: Navigate: this page and this track
On this page
- Progressive Distillation: two teacher moves in one student move (ICLR 2022)
- Consistency Models: the same endpoint along a path (ICML 2023)
- InstaFlow: reflow, then endpoint distillation (ICLR 2024)
- Latent Consistency Models: endpoints in a guided latent flow (2023)
- Adversarial Diffusion Distillation: a teacher and a discriminator (ECCV 2024)
- DMD: matching a teacher-defined distribution (CVPR 2024)
- DMD2: tracking the student and using real data (NeurIPS 2024)
Deep Learning for Computer Vision
Challenge
Endpoint distillation ties a student to the teacher’s image for each random seed. DMD instead seeks agreement between the distributions of their outputs [16]. This allows a student to use a different noise–image map while still generating the desired collection of images.
A distribution objective is harder to implement than a paired image loss. The generator has no tractable image density, and its distribution changes every time its parameters are updated. A fixed teacher score alone does not describe that changing density and can encourage movement toward a limited set of target modes.
Method

Figure source: [16], arXiv:2311.18828v4, PDF page 3. Original PDF.
Generator. Initialize a one-step generator from a pretrained denoiser. Given random input and optional text/class condition , it produces . The output has the teacher’s image or latent representation. Randomness is supplied by ; deterministic evaluation of does not imply deterministic outputs across random seeds.
Noised comparison space. Add noise . Comparing noised distributions avoids clean-support singularities and lets diffusion denoisers supply scores. The selected time distribution and weighting control which scales of discrepancy receive emphasis. Near-clean and high-noise comparisons contain different information.
Target score. A fixed pretrained diffusion model estimates the teacher-defined target score . Calling this the “real score” in a diagram does not make it exact access to the true image distribution. It reflects the quality, conditioning and calibration of the pretrained teacher.
Student score. A second diffusion denoiser is trained on the current generator’s samples, using known added noise or clean-sample targets. It estimates . Its parameters change as the generator distribution changes. The generator update uses a stopped-gradient difference of the two scores as a direction back through ; it need not backpropagate a full Jacobian through the teacher denoiser.
Distribution gradient. Under the noised-distribution construction, the practical update is proportional to . Here normalizes or weights noise-level contributions. The difference includes a correction for probability mass already supplied by the student. It is not equivalent to simply maximizing teacher likelihood for every generated image.
Regression regularizer. Original DMD also prepares noise–image pairs using a deterministic teacher sampler. A regression/perceptual loss pulls selected generator outputs toward those endpoints. This preserves coverage and stabilizes training while the learned student-score estimate is imperfect. The original objective therefore combines distribution freedom with some paired-map regularization; describing it as entirely free of trajectory supervision would omit a major component.
Alternating optimization. Update the student-score network on generated samples and update the generator with the distribution gradient plus endpoint regularizer. These are different optimizers and targets. The teacher remains fixed. At inference only the generator and any representation decoder are needed; the auxiliary score network is a training expense.
Results
The ImageNet-64 comparison reports one-step FID 2.62. The zero-shot COCO-30k text-to-image table reports 512-resolution FID 11.49 for a 0.09-second operating point, while the measured teacher row is 8.78 at 2.59 seconds. These are the paper’s conditions and guidance choices, not a hardware-independent latency ratio. The student does not match every teacher score merely because it is much faster.
The paper separately studies a higher-guidance training setting. Those rows should retain their guidance scale and initialization; a score from one setting cannot be attached to a latency or prompt score from another. Conditional quality and distribution quality can move differently when guidance changes.
Ablations
Table 2 isolates the losses. On CIFAR-10, removing distribution matching gives FID 3.82, removing regression gives 5.58, and the complete method gives 2.66. On ImageNet the corresponding values are 9.21, 5.61 and 2.62. Thus both signals contribute under that training recipe.
Alternative timestep weights give CIFAR FIDs 3.60 and 3.71, compared with 2.66 for the selected weighting. This is evidence that gradient scale across noise levels matters. Visual comparisons also show reduced diversity without regression. They do not prove that the true ideal distribution objective requires regression; the practical score approximation and update dynamics are part of the observed failure. DMD2 investigates that distinction [19].
Conclusion
A teacher score and a learned student score provide a useful distribution-matching direction for a one-step generator. Endpoint regression stabilizes the original implementation, but also constrains some of the freedom the distribution objective was intended to provide.
Problems
A small generator loss does not independently verify that its auxiliary score is accurate. Monitor the changing generated distribution and the auxiliary denoising fit separately. The auxiliary score can lag behind the moving generator and bias its gradient. Teacher-score errors propagate to the target, and finite-noise comparisons do not fully certify clean-distribution coverage. Offline endpoint pairs cost storage and teacher sampling. Prompt conditioning and guidance must be matched in both score networks. DMD2 removes the endpoint dataset while explicitly addressing score-tracking stability and adding real-image supervision.
References
- Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, B. Poole, “Score-Based Generative Modeling through Stochastic Differential Equations,” ICLR, 2021. paper
- Y. Lipman, M. Havasi, P. Holderrieth, et al., “Flow Matching Guide and Code,” arXiv tutorial, 2024. paper
- T. Karras, M. Aittala, T. Aila, S. Laine, “Elucidating the Design Space of Diffusion-Based Generative Models,” NeurIPS, 2022. source
- X. Fan, Z. Wu, H. Wu, L. Cao, “A Survey on Pre-Trained Diffusion Model Distillations,” arXiv preprint, version 2, 2025. paper
- J. Song, C. Meng, S. Ermon, “Denoising Diffusion Implicit Models,” ICLR, 2021. paper
- C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li, J. Zhu, “DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps,” NeurIPS, 2022. paper
- T. Salimans, J. Ho, “Progressive Distillation for Fast Sampling of Diffusion Models,” ICLR, 2022. paper
- Y. Song, P. Dhariwal, M. Chen, I. Sutskever, “Consistency Models,” ICML, 2023. paper
- Y. Song, P. Dhariwal, “Improved Techniques for Training Consistency Models,” ICLR, 2024. paper
- S. Luo, Y. Tan, L. Huang, J. Li, H. Zhao, “Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference,” arXiv preprint, 2023. paper
- R. Rombach, A. Blattmann, D. Lorenz, P. Esser, B. Ommer, “High-Resolution Image Synthesis with Latent Diffusion Models,” CVPR, 2022. source
- X. Liu, C. Gong, Q. Liu, “Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow,” ICLR, 2023. paper
- X. Liu, X. Zhang, J. Ma, J. Peng, Q. Liu, “InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation,” ICLR, 2024. paper
- A. Sauer, D. Lorenz, A. Blattmann, R. Rombach, “Adversarial Diffusion Distillation,” ECCV, 2024. paper
- A. Sauer, F. Boesel, T. Dockhorn, A. Blattmann, P. Esser, R. Rombach, “Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion Distillation,” SIGGRAPH Asia Conference Papers, 2024. paper · published version
- T. Yin, M. Gharbi, R. Zhang, E. Shechtman, F. Durand, W. T. Freeman, T. Park, “One-step Diffusion with Distribution Matching Distillation,” CVPR, 2024. paper
- T. H. Nguyen, A. Tran, “SwiftBrush: One-Step Text-to-Image Diffusion Model with Variational Score Distillation,” CVPR, 2024. paper
- M. Zhou, H. Zheng, Z. Wang, M. Yin, H. Huang, “Score identity Distillation: Exponentially Fast Distillation of Pretrained Diffusion Models for One-Step Generation,” ICML, 2024. paper
- T. Yin, M. Gharbi, T. Park, R. Zhang, E. Shechtman, F. Durand, W. T. Freeman, “Improved Distribution Matching Distillation for Fast Image Synthesis,” NeurIPS, 2024. paper
- K. Frans, D. Hafner, S. Levine, P. Abbeel, “One Step Diffusion via Shortcut Models,” ICLR, 2025. paper
- Z. Geng, M. Deng, X. Bai, J. Z. Kolter, K. He, “Mean Flows for One-step Generative Modeling,” NeurIPS, 2025. paper
- M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, S. Hochreiter, “GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium,” NeurIPS, 2017. source
- A. Radford, J. W. Kim, C. Hallacy, et al., “Learning Transferable Visual Models From Natural Language Supervision,” ICML, 2021. paper
- D. Ghosh, H. Hajishirzi, L. Schmidt, “GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment,” NeurIPS Datasets and Benchmarks, 2023. paper
- A. Krizhevsky, V. Nair and G. Hinton, “CIFAR-10 and CIFAR-100 datasets,” official dataset page; companion technical report: A. Krizhevsky, “Learning Multiple Layers of Features from Tiny Images,” 2009. dataset and report
- J. Deng, W. Dong, R. Socher, L. Li, Kai Li, Li Fei-Fei, “ImageNet: A large-scale hierarchical image database,” CVPR, 2009. doi
- T. Lin, M. Maire, S. Belongie, et al., “Microsoft COCO: Common Objects in Context,” ECCV, 2014. paper
- L. Yang, Z. Zhang, Y. Song, et al., “Diffusion Models: A Comprehensive Survey of Methods and Applications,” ACM Computing Surveys, 2023. paper
- Z. Yu, J. Yuan, H. Yang, et al., “DMAD: Distribution Matching as Adversarial Distillation for Fast Visual Generation,” arXiv preprint, 2026. paper
- R. Zhang, P. Isola, A. A. Efros, E. Shechtman and O. Wang, “The Unreasonable Effectiveness of Deep Features as a Perceptual Metric,” in Proc. IEEE/CVF CVPR, 2018, pp. 586–595. doi · arXiv
- C. Schuhmann, R. Beaumont, R. Vencu, et al., “LAION-5B: An open large-scale dataset for training next generation image-text models,” NeurIPS Datasets and Benchmarks, 2022. paper