Full search page →

DMD2: tracking the student and using real data (NeurIPS 2024)

Optional · paper guide 7 of 7 for DLCV 12B · One-Step Generation. Citation numbers refer to the reference list at the bottom of this page.

Navigate: this page and this track: Navigate: this page and this track

On this page

Supplementary notes · 2

  1. Instantaneous velocity versus interval velocity
  2. Three meanings of teacher-free

Paper guides · 7

  1. Progressive Distillation: two teacher moves in one student move (ICLR 2022)
  2. Consistency Models: the same endpoint along a path (ICML 2023)
  3. InstaFlow: reflow, then endpoint distillation (ICLR 2024)
  4. Latent Consistency Models: endpoints in a guided latent flow (2023)
  5. Adversarial Diffusion Distillation: a teacher and a discriminator (ECCV 2024)
  6. DMD: matching a teacher-defined distribution (CVPR 2024)
  7. DMD2: tracking the student and using real data (NeurIPS 2024)

Deep Learning for Computer Vision

  1. 02 · Deep Image Classification
View the whole course tree →

Challenge

DMD’s endpoint regularizer stabilizes training but requires expensive teacher-generated pairs and partly binds the student to a selected teacher map. DMD2 asks whether distribution matching can work without that dataset, and why simply deleting the regularizer fails [19]. It identifies inaccurate tracking of the changing student distribution as a central problem.

A second issue is teacher quality. A frozen “real” score is a pretrained model’s estimate, not the true data density. If it contains systematic errors, matching it alone cannot supply corrective evidence from actual images. A third issue appears for multi-step students: their intermediate inference states can differ from noisy training states constructed by a teacher-style forward process.

Method

The generated-sample score model tracks the student, while real data also supplies adversarial training; the diagram separates those learning signals from generation.
Original paper figure 3 — The generated-sample score model tracks the student, while real data also supplies adversarial training; the diagram separates those learning signals from generation.

Figure source: [19], arXiv:2405.14867v2, PDF page 6. Original PDF.

Remove paired endpoint regression. The generator is no longer trained against a precomputed image for each input noise seed. The fixed target score and trainable student score remain. This removes offline multi-step target construction and permits a different coupling, but it makes the quality of the score difference more important.

Two time scales. Train the auxiliary student-score estimator more often than the generator, using five score updates per generator update in the reported configuration. The goal is to let it track the current output distribution before that distribution moves again. This costs training compute and does not make the auxiliary score exact. It addresses an optimization imbalance rather than adding an inference step.

Real-image discriminator. Add a classification head to the auxiliary denoiser’s bottleneck features and train an adversarial objective using real and generated images at sampled noise levels. The shared representation supplies a compact discriminator rather than a completely separate large image network. Real images provide a supervision source that is not restricted to the teacher’s estimated density. The generator receives both distribution-matching and adversarial gradients.

Multi-step student. For a student queried at several noise levels, initialize it from a pretrained denoiser and let its time-conditioned predictions generate a sequence. During training, simulate the student’s own preceding steps to construct the next input. This “backward simulation” reduces mismatch between states made by a forward corruption process and states visited by the deployed student. It is a training-data construction for the student’s trajectory; it does not restore the removed teacher-pair regression dataset.

Guidance and representation. Target-score conditioning and generation prompts must agree. Distilling a guided teacher can bake guidance behavior into the short sampler. The latent decoder remains when the generator operates in image latents. Teacher, auxiliary denoiser and discriminator heads are required for training; inference can discard those auxiliary paths.

Why the parts differ. More auxiliary updates improve tracking; real-data adversarial training supplies a different target signal; student-state simulation changes the input distribution for a multi-step generator. None is redundant with the others. In particular, the tracking change alone cannot remove a systematic error in the teacher’s target score, and adversarial training alone does not enforce the diffusion-derived distribution objective.

Results

On ImageNet-64, the paper reports one-step FID 1.51, and 1.28 with longer training. The teacher’s reported ODE-sampling FID is 2.32, while its SDE result is 1.36. “Surpasses the teacher” therefore needs the sampling comparator attached: beating one teacher sampler’s FID is not beating every teacher operating point or every quality measure.

With the SDXL backbone on 10,000 COCO-2014 prompts, Table 2 reports one-step FID 19.01 and CLIP score 0.336, and four-step FID 19.32 and CLIP score 0.332. Patch FID improves from 26.98 to 20.86 at four steps. Different metrics therefore favor different budgets. The four-step teacher-comparison claims must not be silently reassigned to a one-step row, and the 10k protocol is distinct from a 30k evaluation.

Ablations

On ImageNet, original DMD gives FID 2.62. Deleting regression worsens it to 3.48. Adding the different update frequencies gives 2.61; adding the real-data GAN objective gives 1.51. GAN-only training gives 2.56, and the GAN variant with the update-frequency change gives 2.52. The complete combination improves over those individual components under the tested setting.

For the four-step SDXL student, removing GAN training gives FID 26.90 and patch FID 27.66; the complete model gives 19.32 and 20.86. Removing distribution matching gives a superficially better full-image FID 13.77 but worse patch FID 27.96 and CLIP score 0.307 rather than 0.332. This is a particularly useful warning against one-number ranking. Removing student-state simulation gives FID 20.66 and patch FID 24.21, showing its role in this multi-step setting.

Conclusion

Distribution-matching distillation needs the student’s moving distribution to be tracked accurately. More balanced optimization removes the original endpoint regularizer, while real images and student-generated training states address different sources of error.

Problems

Training still involves large pretrained models, auxiliary updates and real-image data. Better benchmark scores do not certify rare-mode coverage or compositional correctness. Student-state simulation increases training work, and the best budget depends on the metric. The method reduces deployment sampling cost, but it does not make training cheap or eliminate all teacher biases. Later adversarial distribution approaches explore different ways to avoid auxiliary-score tracking [29].

References

  1. Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, B. Poole, “Score-Based Generative Modeling through Stochastic Differential Equations,” ICLR, 2021. paper
  2. Y. Lipman, M. Havasi, P. Holderrieth, et al., “Flow Matching Guide and Code,” arXiv tutorial, 2024. paper
  3. T. Karras, M. Aittala, T. Aila, S. Laine, “Elucidating the Design Space of Diffusion-Based Generative Models,” NeurIPS, 2022. source
  4. X. Fan, Z. Wu, H. Wu, L. Cao, “A Survey on Pre-Trained Diffusion Model Distillations,” arXiv preprint, version 2, 2025. paper
  5. J. Song, C. Meng, S. Ermon, “Denoising Diffusion Implicit Models,” ICLR, 2021. paper
  6. C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li, J. Zhu, “DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps,” NeurIPS, 2022. paper
  7. T. Salimans, J. Ho, “Progressive Distillation for Fast Sampling of Diffusion Models,” ICLR, 2022. paper
  8. Y. Song, P. Dhariwal, M. Chen, I. Sutskever, “Consistency Models,” ICML, 2023. paper
  9. Y. Song, P. Dhariwal, “Improved Techniques for Training Consistency Models,” ICLR, 2024. paper
  10. S. Luo, Y. Tan, L. Huang, J. Li, H. Zhao, “Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference,” arXiv preprint, 2023. paper
  11. R. Rombach, A. Blattmann, D. Lorenz, P. Esser, B. Ommer, “High-Resolution Image Synthesis with Latent Diffusion Models,” CVPR, 2022. source
  12. X. Liu, C. Gong, Q. Liu, “Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow,” ICLR, 2023. paper
  13. X. Liu, X. Zhang, J. Ma, J. Peng, Q. Liu, “InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation,” ICLR, 2024. paper
  14. A. Sauer, D. Lorenz, A. Blattmann, R. Rombach, “Adversarial Diffusion Distillation,” ECCV, 2024. paper
  15. A. Sauer, F. Boesel, T. Dockhorn, A. Blattmann, P. Esser, R. Rombach, “Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion Distillation,” SIGGRAPH Asia Conference Papers, 2024. paper · published version
  16. T. Yin, M. Gharbi, R. Zhang, E. Shechtman, F. Durand, W. T. Freeman, T. Park, “One-step Diffusion with Distribution Matching Distillation,” CVPR, 2024. paper
  17. T. H. Nguyen, A. Tran, “SwiftBrush: One-Step Text-to-Image Diffusion Model with Variational Score Distillation,” CVPR, 2024. paper
  18. M. Zhou, H. Zheng, Z. Wang, M. Yin, H. Huang, “Score identity Distillation: Exponentially Fast Distillation of Pretrained Diffusion Models for One-Step Generation,” ICML, 2024. paper
  19. T. Yin, M. Gharbi, T. Park, R. Zhang, E. Shechtman, F. Durand, W. T. Freeman, “Improved Distribution Matching Distillation for Fast Image Synthesis,” NeurIPS, 2024. paper
  20. K. Frans, D. Hafner, S. Levine, P. Abbeel, “One Step Diffusion via Shortcut Models,” ICLR, 2025. paper
  21. Z. Geng, M. Deng, X. Bai, J. Z. Kolter, K. He, “Mean Flows for One-step Generative Modeling,” NeurIPS, 2025. paper
  22. M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, S. Hochreiter, “GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium,” NeurIPS, 2017. source
  23. A. Radford, J. W. Kim, C. Hallacy, et al., “Learning Transferable Visual Models From Natural Language Supervision,” ICML, 2021. paper
  24. D. Ghosh, H. Hajishirzi, L. Schmidt, “GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment,” NeurIPS Datasets and Benchmarks, 2023. paper
  25. A. Krizhevsky, V. Nair and G. Hinton, “CIFAR-10 and CIFAR-100 datasets,” official dataset page; companion technical report: A. Krizhevsky, “Learning Multiple Layers of Features from Tiny Images,” 2009. dataset and report
  26. J. Deng, W. Dong, R. Socher, L. Li, Kai Li, Li Fei-Fei, “ImageNet: A large-scale hierarchical image database,” CVPR, 2009. doi
  27. T. Lin, M. Maire, S. Belongie, et al., “Microsoft COCO: Common Objects in Context,” ECCV, 2014. paper
  28. L. Yang, Z. Zhang, Y. Song, et al., “Diffusion Models: A Comprehensive Survey of Methods and Applications,” ACM Computing Surveys, 2023. paper
  29. Z. Yu, J. Yuan, H. Yang, et al., “DMAD: Distribution Matching as Adversarial Distillation for Fast Visual Generation,” arXiv preprint, 2026. paper
  30. R. Zhang, P. Isola, A. A. Efros, E. Shechtman and O. Wang, “The Unreasonable Effectiveness of Deep Features as a Perceptual Metric,” in Proc. IEEE/CVF CVPR, 2018, pp. 586–595. doi · arXiv
  31. C. Schuhmann, R. Beaumont, R. Vencu, et al., “LAION-5B: An open large-scale dataset for training next generation image-text models,” NeurIPS Datasets and Benchmarks, 2022. paper