Full search page →

DMD: matching a teacher-defined distribution (CVPR 2024)

Optional · paper guide 6 of 7 for DLCV 12B · One-Step Generation. Citation numbers refer to the reference list at the bottom of this page.

Navigate: this page and this track: Navigate: this page and this track

On this page

Supplementary notes · 2

  1. Instantaneous velocity versus interval velocity
  2. Three meanings of teacher-free

Paper guides · 7

  1. Progressive Distillation: two teacher moves in one student move (ICLR 2022)
  2. Consistency Models: the same endpoint along a path (ICML 2023)
  3. InstaFlow: reflow, then endpoint distillation (ICLR 2024)
  4. Latent Consistency Models: endpoints in a guided latent flow (2023)
  5. Adversarial Diffusion Distillation: a teacher and a discriminator (ECCV 2024)
  6. DMD: matching a teacher-defined distribution (CVPR 2024)
  7. DMD2: tracking the student and using real data (NeurIPS 2024)

Deep Learning for Computer Vision

  1. 02 · Deep Image Classification
View the whole course tree →

Challenge

Endpoint distillation ties a student to the teacher’s image for each random seed. DMD instead seeks agreement between the distributions of their outputs [16]. This allows a student to use a different noise–image map while still generating the desired collection of images.

A distribution objective is harder to implement than a paired image loss. The generator has no tractable image density, and its distribution changes every time its parameters are updated. A fixed teacher score alone does not describe that changing density and can encourage movement toward a limited set of target modes.

Method

Compare the teacher-defined real score with the score fitted to generated samples; their discrepancy supplies distribution-matching gradients alongside the regression term.
Original paper figure 2 — Compare the teacher-defined real score with the score fitted to generated samples; their discrepancy supplies distribution-matching gradients alongside the regression term.

Figure source: [16], arXiv:2311.18828v4, PDF page 3. Original PDF.

Generator. Initialize a one-step generator from a pretrained denoiser. Given random input zz and optional text/class condition cc, it produces x=Gθ(z,c)x=G_\theta(z,c). The output has the teacher’s image or latent representation. Randomness is supplied by zz; deterministic evaluation of GθG_\theta does not imply deterministic outputs across random seeds.

Noised comparison space. Add noise xt=αtx+σtϵx_t=\alpha_tx+\sigma_t\epsilon. Comparing noised distributions avoids clean-support singularities and lets diffusion denoisers supply scores. The selected time distribution and weighting control which scales of discrepancy receive emphasis. Near-clean and high-noise comparisons contain different information.

Target score. A fixed pretrained diffusion model estimates the teacher-defined target score starget(xt,t,c)s_{\mathrm{target}}(x_t,t,c). Calling this the “real score” in a diagram does not make it exact access to the true image distribution. It reflects the quality, conditioning and calibration of the pretrained teacher.

Student score. A second diffusion denoiser is trained on the current generator’s samples, using known added noise or clean-sample targets. It estimates sstudents_{\mathrm{student}}. Its parameters change as the generator distribution changes. The generator update uses a stopped-gradient difference of the two scores as a direction back through GθG_\theta; it need not backpropagate a full Jacobian through the teacher denoiser.

Distribution gradient. Under the noised-distribution construction, the practical update is proportional to E[w(t)αt(sstudent−starget)∂θGθ]\mathbb E[w(t)\alpha_t(s_{\mathrm{student}}-s_{\mathrm{target}})\partial_\theta G_\theta]. Here w(t)w(t) normalizes or weights noise-level contributions. The difference includes a correction for probability mass already supplied by the student. It is not equivalent to simply maximizing teacher likelihood for every generated image.

Regression regularizer. Original DMD also prepares noise–image pairs using a deterministic teacher sampler. A regression/perceptual loss pulls selected generator outputs toward those endpoints. This preserves coverage and stabilizes training while the learned student-score estimate is imperfect. The original objective therefore combines distribution freedom with some paired-map regularization; describing it as entirely free of trajectory supervision would omit a major component.

Alternating optimization. Update the student-score network on generated samples and update the generator with the distribution gradient plus endpoint regularizer. These are different optimizers and targets. The teacher remains fixed. At inference only the generator and any representation decoder are needed; the auxiliary score network is a training expense.

Results

The ImageNet-64 comparison reports one-step FID 2.62. The zero-shot COCO-30k text-to-image table reports 512-resolution FID 11.49 for a 0.09-second operating point, while the measured teacher row is 8.78 at 2.59 seconds. These are the paper’s conditions and guidance choices, not a hardware-independent latency ratio. The student does not match every teacher score merely because it is much faster.

The paper separately studies a higher-guidance training setting. Those rows should retain their guidance scale and initialization; a score from one setting cannot be attached to a latency or prompt score from another. Conditional quality and distribution quality can move differently when guidance changes.

Ablations

Table 2 isolates the losses. On CIFAR-10, removing distribution matching gives FID 3.82, removing regression gives 5.58, and the complete method gives 2.66. On ImageNet the corresponding values are 9.21, 5.61 and 2.62. Thus both signals contribute under that training recipe.

Alternative timestep weights give CIFAR FIDs 3.60 and 3.71, compared with 2.66 for the selected weighting. This is evidence that gradient scale across noise levels matters. Visual comparisons also show reduced diversity without regression. They do not prove that the true ideal distribution objective requires regression; the practical score approximation and update dynamics are part of the observed failure. DMD2 investigates that distinction [19].

Conclusion

A teacher score and a learned student score provide a useful distribution-matching direction for a one-step generator. Endpoint regression stabilizes the original implementation, but also constrains some of the freedom the distribution objective was intended to provide.

Problems

A small generator loss does not independently verify that its auxiliary score is accurate. Monitor the changing generated distribution and the auxiliary denoising fit separately. The auxiliary score can lag behind the moving generator and bias its gradient. Teacher-score errors propagate to the target, and finite-noise comparisons do not fully certify clean-distribution coverage. Offline endpoint pairs cost storage and teacher sampling. Prompt conditioning and guidance must be matched in both score networks. DMD2 removes the endpoint dataset while explicitly addressing score-tracking stability and adding real-image supervision.

References

  1. Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, B. Poole, “Score-Based Generative Modeling through Stochastic Differential Equations,” ICLR, 2021. paper
  2. Y. Lipman, M. Havasi, P. Holderrieth, et al., “Flow Matching Guide and Code,” arXiv tutorial, 2024. paper
  3. T. Karras, M. Aittala, T. Aila, S. Laine, “Elucidating the Design Space of Diffusion-Based Generative Models,” NeurIPS, 2022. source
  4. X. Fan, Z. Wu, H. Wu, L. Cao, “A Survey on Pre-Trained Diffusion Model Distillations,” arXiv preprint, version 2, 2025. paper
  5. J. Song, C. Meng, S. Ermon, “Denoising Diffusion Implicit Models,” ICLR, 2021. paper
  6. C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li, J. Zhu, “DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps,” NeurIPS, 2022. paper
  7. T. Salimans, J. Ho, “Progressive Distillation for Fast Sampling of Diffusion Models,” ICLR, 2022. paper
  8. Y. Song, P. Dhariwal, M. Chen, I. Sutskever, “Consistency Models,” ICML, 2023. paper
  9. Y. Song, P. Dhariwal, “Improved Techniques for Training Consistency Models,” ICLR, 2024. paper
  10. S. Luo, Y. Tan, L. Huang, J. Li, H. Zhao, “Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference,” arXiv preprint, 2023. paper
  11. R. Rombach, A. Blattmann, D. Lorenz, P. Esser, B. Ommer, “High-Resolution Image Synthesis with Latent Diffusion Models,” CVPR, 2022. source
  12. X. Liu, C. Gong, Q. Liu, “Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow,” ICLR, 2023. paper
  13. X. Liu, X. Zhang, J. Ma, J. Peng, Q. Liu, “InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation,” ICLR, 2024. paper
  14. A. Sauer, D. Lorenz, A. Blattmann, R. Rombach, “Adversarial Diffusion Distillation,” ECCV, 2024. paper
  15. A. Sauer, F. Boesel, T. Dockhorn, A. Blattmann, P. Esser, R. Rombach, “Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion Distillation,” SIGGRAPH Asia Conference Papers, 2024. paper · published version
  16. T. Yin, M. Gharbi, R. Zhang, E. Shechtman, F. Durand, W. T. Freeman, T. Park, “One-step Diffusion with Distribution Matching Distillation,” CVPR, 2024. paper
  17. T. H. Nguyen, A. Tran, “SwiftBrush: One-Step Text-to-Image Diffusion Model with Variational Score Distillation,” CVPR, 2024. paper
  18. M. Zhou, H. Zheng, Z. Wang, M. Yin, H. Huang, “Score identity Distillation: Exponentially Fast Distillation of Pretrained Diffusion Models for One-Step Generation,” ICML, 2024. paper
  19. T. Yin, M. Gharbi, T. Park, R. Zhang, E. Shechtman, F. Durand, W. T. Freeman, “Improved Distribution Matching Distillation for Fast Image Synthesis,” NeurIPS, 2024. paper
  20. K. Frans, D. Hafner, S. Levine, P. Abbeel, “One Step Diffusion via Shortcut Models,” ICLR, 2025. paper
  21. Z. Geng, M. Deng, X. Bai, J. Z. Kolter, K. He, “Mean Flows for One-step Generative Modeling,” NeurIPS, 2025. paper
  22. M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, S. Hochreiter, “GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium,” NeurIPS, 2017. source
  23. A. Radford, J. W. Kim, C. Hallacy, et al., “Learning Transferable Visual Models From Natural Language Supervision,” ICML, 2021. paper
  24. D. Ghosh, H. Hajishirzi, L. Schmidt, “GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment,” NeurIPS Datasets and Benchmarks, 2023. paper
  25. A. Krizhevsky, V. Nair and G. Hinton, “CIFAR-10 and CIFAR-100 datasets,” official dataset page; companion technical report: A. Krizhevsky, “Learning Multiple Layers of Features from Tiny Images,” 2009. dataset and report
  26. J. Deng, W. Dong, R. Socher, L. Li, Kai Li, Li Fei-Fei, “ImageNet: A large-scale hierarchical image database,” CVPR, 2009. doi
  27. T. Lin, M. Maire, S. Belongie, et al., “Microsoft COCO: Common Objects in Context,” ECCV, 2014. paper
  28. L. Yang, Z. Zhang, Y. Song, et al., “Diffusion Models: A Comprehensive Survey of Methods and Applications,” ACM Computing Surveys, 2023. paper
  29. Z. Yu, J. Yuan, H. Yang, et al., “DMAD: Distribution Matching as Adversarial Distillation for Fast Visual Generation,” arXiv preprint, 2026. paper
  30. R. Zhang, P. Isola, A. A. Efros, E. Shechtman and O. Wang, “The Unreasonable Effectiveness of Deep Features as a Perceptual Metric,” in Proc. IEEE/CVF CVPR, 2018, pp. 586–595. doi · arXiv
  31. C. Schuhmann, R. Beaumont, R. Vencu, et al., “LAION-5B: An open large-scale dataset for training next generation image-text models,” NeurIPS Datasets and Benchmarks, 2022. paper