Full search page →

Progressive Distillation: two teacher moves in one student move (ICLR 2022)

Optional · paper guide 1 of 7 for DLCV 12B · One-Step Generation. Citation numbers refer to the reference list at the bottom of this page.

Navigate: this page and this track: Navigate: this page and this track

On this page

Supplementary notes · 2

  1. Instantaneous velocity versus interval velocity
  2. Three meanings of teacher-free

Paper guides · 7

  1. Progressive Distillation: two teacher moves in one student move (ICLR 2022)
  2. Consistency Models: the same endpoint along a path (ICML 2023)
  3. InstaFlow: reflow, then endpoint distillation (ICLR 2024)
  4. Latent Consistency Models: endpoints in a guided latent flow (2023)
  5. Adversarial Diffusion Distillation: a teacher and a discriminator (ECCV 2024)
  6. DMD: matching a teacher-defined distribution (CVPR 2024)
  7. DMD2: tracking the student and using real data (NeurIPS 2024)

Deep Learning for Computer Vision

  1. 02 · Deep Image Classification
View the whole course tree →

Challenge

An accurate diffusion sampler can require many network evaluations. Taking fewer updates with unchanged predictions introduces discretization error. Salimans and Ho instead ask a student to learn the effect of two deterministic teacher updates in one update [7]. The method amortizes integration into training: it does not claim that an ordinary denoising prediction is already a good full-sample endpoint.

A useful distinction motivates the target. Given a noisy observation, several clean training images can be plausible, so ordinary squared-error denoising predicts an average. Given the noisy state and a deterministic teacher sampler, the teacher’s endpoint is fixed. Regressing that endpoint avoids averaging unrelated clean candidates, although it inherits the teacher’s particular sampling map.

Method

A student learns the endpoint of two teacher steps, then becomes the next teacher; repeated halving reduces inference steps while requiring successive training stages.
Original paper figure 1 — A student learns the endpoint of two teacher steps, then becomes the next teacher; repeated halving reduces inference steps while requiring successive training stages.

Figure source: [7], arXiv:2202.00512v2, PDF page 2. Original PDF.

Teacher and schedule. Start with a pretrained diffusion model and a deterministic DDIM sampler [5]. The state is zt=αtx+σtϵz_t=\alpha_t x+\sigma_t\epsilon, where xx is data, ϵ\epsilon standard Gaussian noise, and αt,σt\alpha_t,\sigma_t a signal/noise schedule. The student is initialized from the teacher with the same architecture. Times are sampled on the student’s discrete grid; the terminal state uses zero signal-to-noise ratio, matching pure-noise generation.

Two teacher steps. For an interval from tt to ss, first run the teacher to the midpoint and then to ss, producing zs(2)z_s^{(2)}. These operations are training-target construction. At deployment the student is not required to run them. The state and target retain the image or latent tensor shape; no extra sequence of intermediate states is supplied as a student input.

Invert the student update. A single DDIM step is affine in its clean-image prediction. The required target is

x~=zs(2)−(σs/σt)ztαs−(σs/σt)αt.\widetilde x=\frac{z_s^{(2)}-(\sigma_s/\sigma_t)z_t}{\alpha_s-(\sigma_s/\sigma_t)\alpha_t}.

Here ztz_t is the shared starting state, zs(2)z_s^{(2)} the teacher endpoint, and s<ts<t a cleaner time. Predicting x~\widetilde x reproduces those two teacher moves through the one-step DDIM formula. The student minimizes a schedule-weighted squared prediction error. This is not simply training against the original clean sample or the teacher’s first prediction.

Prediction parameterization. Direct noise prediction becomes awkward at zero signal because conversion to a clean image divides by αt\alpha_t. The paper studies clean-image prediction, combined clean/noise prediction and velocity prediction, together with loss weights based on signal-to-noise ratio. For a variance-preserving schedule, a velocity target can be written v=αtϵ−σtxv=\alpha_t\epsilon-\sigma_t x. This is a rotation of the clean/noise coordinates; its conversion properties help avoid a singular clean reconstruction at a noise-only boundary. Parameterization and loss weighting jointly determine which noise levels dominate training.

Progression. After fitting an NN-step student, use it as the next teacher and train an N/2N/2-step student. Repeat until the desired budget is reached. Each stage only has to learn a factor-of-two compression, but every stage costs training and can transfer approximation errors to the next one. This is a curriculum over discretizations rather than a new forward corruption process.

Results

Table 2 reports unconditional CIFAR-10 FID [25], [22]: 9.12 at one model evaluation, 4.51 at two, 3.00 at four and 2.57 at eight. The inference computation falls, while the training stages supply the information that a coarse solver would otherwise miss. These are operating points of the progressively distilled family, not one model evaluated with an arbitrary external schedule. The four-step result preserves much more of the original model’s sample quality than an unmodified sampler at a similarly small budget.

The paper also studies larger-resolution generation, but its CIFAR result should not be read as proof that one-step text-to-image generation has the same trade-off. The task, architecture, conditioning and evaluation distribution differ. Later papers may report different progressive-distillation baselines because they reimplement or retune the method; attach a baseline to the paper that measured it.

Ablations

Table 1 compares parameterizations before distillation on unconditional CIFAR-10. With clean-image prediction and truncated-SNR weighting, its deterministic sampler gives FID 2.51; noise prediction with the ordinary SNR weighting gives 2.91. The noise-prediction/truncated-SNR combination diverges. These results support stable output/weighting choices, not a claim that 2.51 is the one-step distilled score.

The evaluation-count sweep above is a second important study: fewer stages of inference progressively worsen sample quality, so compression has a measurable limit. The paper also compares numerical integrators and studies prediction behavior across noise levels. It does not establish that repeatedly halving indefinitely can preserve an exact teacher distribution with a fixed finite network.

Conclusion

Learning a finite teacher transition is more effective than asking an unchanged denoiser to take a very large step. The target is analytically constructed by inverting the student’s deterministic update, and progressive halving provides a practical training curriculum.

Problems

The procedure requires a teacher, several training stages and compatible schedules. It preserves a selected deterministic coupling, including that coupling’s defects. Very small budgets still lose quality, and changes to guidance or conditioning after training need care. Consistency models supply a different endpoint-learning objective; distribution matching later relaxes the demand to reproduce each teacher trajectory [8], [16].

References

  1. Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, B. Poole, “Score-Based Generative Modeling through Stochastic Differential Equations,” ICLR, 2021. paper
  2. Y. Lipman, M. Havasi, P. Holderrieth, et al., “Flow Matching Guide and Code,” arXiv tutorial, 2024. paper
  3. T. Karras, M. Aittala, T. Aila, S. Laine, “Elucidating the Design Space of Diffusion-Based Generative Models,” NeurIPS, 2022. source
  4. X. Fan, Z. Wu, H. Wu, L. Cao, “A Survey on Pre-Trained Diffusion Model Distillations,” arXiv preprint, version 2, 2025. paper
  5. J. Song, C. Meng, S. Ermon, “Denoising Diffusion Implicit Models,” ICLR, 2021. paper
  6. C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li, J. Zhu, “DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps,” NeurIPS, 2022. paper
  7. T. Salimans, J. Ho, “Progressive Distillation for Fast Sampling of Diffusion Models,” ICLR, 2022. paper
  8. Y. Song, P. Dhariwal, M. Chen, I. Sutskever, “Consistency Models,” ICML, 2023. paper
  9. Y. Song, P. Dhariwal, “Improved Techniques for Training Consistency Models,” ICLR, 2024. paper
  10. S. Luo, Y. Tan, L. Huang, J. Li, H. Zhao, “Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference,” arXiv preprint, 2023. paper
  11. R. Rombach, A. Blattmann, D. Lorenz, P. Esser, B. Ommer, “High-Resolution Image Synthesis with Latent Diffusion Models,” CVPR, 2022. source
  12. X. Liu, C. Gong, Q. Liu, “Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow,” ICLR, 2023. paper
  13. X. Liu, X. Zhang, J. Ma, J. Peng, Q. Liu, “InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation,” ICLR, 2024. paper
  14. A. Sauer, D. Lorenz, A. Blattmann, R. Rombach, “Adversarial Diffusion Distillation,” ECCV, 2024. paper
  15. A. Sauer, F. Boesel, T. Dockhorn, A. Blattmann, P. Esser, R. Rombach, “Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion Distillation,” SIGGRAPH Asia Conference Papers, 2024. paper · published version
  16. T. Yin, M. Gharbi, R. Zhang, E. Shechtman, F. Durand, W. T. Freeman, T. Park, “One-step Diffusion with Distribution Matching Distillation,” CVPR, 2024. paper
  17. T. H. Nguyen, A. Tran, “SwiftBrush: One-Step Text-to-Image Diffusion Model with Variational Score Distillation,” CVPR, 2024. paper
  18. M. Zhou, H. Zheng, Z. Wang, M. Yin, H. Huang, “Score identity Distillation: Exponentially Fast Distillation of Pretrained Diffusion Models for One-Step Generation,” ICML, 2024. paper
  19. T. Yin, M. Gharbi, T. Park, R. Zhang, E. Shechtman, F. Durand, W. T. Freeman, “Improved Distribution Matching Distillation for Fast Image Synthesis,” NeurIPS, 2024. paper
  20. K. Frans, D. Hafner, S. Levine, P. Abbeel, “One Step Diffusion via Shortcut Models,” ICLR, 2025. paper
  21. Z. Geng, M. Deng, X. Bai, J. Z. Kolter, K. He, “Mean Flows for One-step Generative Modeling,” NeurIPS, 2025. paper
  22. M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, S. Hochreiter, “GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium,” NeurIPS, 2017. source
  23. A. Radford, J. W. Kim, C. Hallacy, et al., “Learning Transferable Visual Models From Natural Language Supervision,” ICML, 2021. paper
  24. D. Ghosh, H. Hajishirzi, L. Schmidt, “GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment,” NeurIPS Datasets and Benchmarks, 2023. paper
  25. A. Krizhevsky, V. Nair and G. Hinton, “CIFAR-10 and CIFAR-100 datasets,” official dataset page; companion technical report: A. Krizhevsky, “Learning Multiple Layers of Features from Tiny Images,” 2009. dataset and report
  26. J. Deng, W. Dong, R. Socher, L. Li, Kai Li, Li Fei-Fei, “ImageNet: A large-scale hierarchical image database,” CVPR, 2009. doi
  27. T. Lin, M. Maire, S. Belongie, et al., “Microsoft COCO: Common Objects in Context,” ECCV, 2014. paper
  28. L. Yang, Z. Zhang, Y. Song, et al., “Diffusion Models: A Comprehensive Survey of Methods and Applications,” ACM Computing Surveys, 2023. paper
  29. Z. Yu, J. Yuan, H. Yang, et al., “DMAD: Distribution Matching as Adversarial Distillation for Fast Visual Generation,” arXiv preprint, 2026. paper
  30. R. Zhang, P. Isola, A. A. Efros, E. Shechtman and O. Wang, “The Unreasonable Effectiveness of Deep Features as a Perceptual Metric,” in Proc. IEEE/CVF CVPR, 2018, pp. 586–595. doi · arXiv
  31. C. Schuhmann, R. Beaumont, R. Vencu, et al., “LAION-5B: An open large-scale dataset for training next generation image-text models,” NeurIPS Datasets and Benchmarks, 2022. paper