Progressive Distillation: two teacher moves in one student move (ICLR 2022)
Optional · paper guide 1 of 7 for DLCV 12B · One-Step Generation. Citation numbers refer to the reference list at the bottom of this page.
Navigate: this page and this track: Navigate: this page and this track
On this page
- Progressive Distillation: two teacher moves in one student move (ICLR 2022)
- Consistency Models: the same endpoint along a path (ICML 2023)
- InstaFlow: reflow, then endpoint distillation (ICLR 2024)
- Latent Consistency Models: endpoints in a guided latent flow (2023)
- Adversarial Diffusion Distillation: a teacher and a discriminator (ECCV 2024)
- DMD: matching a teacher-defined distribution (CVPR 2024)
- DMD2: tracking the student and using real data (NeurIPS 2024)
Deep Learning for Computer Vision
Challenge
An accurate diffusion sampler can require many network evaluations. Taking fewer updates with unchanged predictions introduces discretization error. Salimans and Ho instead ask a student to learn the effect of two deterministic teacher updates in one update [7]. The method amortizes integration into training: it does not claim that an ordinary denoising prediction is already a good full-sample endpoint.
A useful distinction motivates the target. Given a noisy observation, several clean training images can be plausible, so ordinary squared-error denoising predicts an average. Given the noisy state and a deterministic teacher sampler, the teacher’s endpoint is fixed. Regressing that endpoint avoids averaging unrelated clean candidates, although it inherits the teacher’s particular sampling map.
Method

Figure source: [7], arXiv:2202.00512v2, PDF page 2. Original PDF.
Teacher and schedule. Start with a pretrained diffusion model and a deterministic DDIM sampler [5]. The state is , where is data, standard Gaussian noise, and a signal/noise schedule. The student is initialized from the teacher with the same architecture. Times are sampled on the student’s discrete grid; the terminal state uses zero signal-to-noise ratio, matching pure-noise generation.
Two teacher steps. For an interval from to , first run the teacher to the midpoint and then to , producing . These operations are training-target construction. At deployment the student is not required to run them. The state and target retain the image or latent tensor shape; no extra sequence of intermediate states is supplied as a student input.
Invert the student update. A single DDIM step is affine in its clean-image prediction. The required target is
Here is the shared starting state, the teacher endpoint, and a cleaner time. Predicting reproduces those two teacher moves through the one-step DDIM formula. The student minimizes a schedule-weighted squared prediction error. This is not simply training against the original clean sample or the teacher’s first prediction.
Prediction parameterization. Direct noise prediction becomes awkward at zero signal because conversion to a clean image divides by . The paper studies clean-image prediction, combined clean/noise prediction and velocity prediction, together with loss weights based on signal-to-noise ratio. For a variance-preserving schedule, a velocity target can be written . This is a rotation of the clean/noise coordinates; its conversion properties help avoid a singular clean reconstruction at a noise-only boundary. Parameterization and loss weighting jointly determine which noise levels dominate training.
Progression. After fitting an -step student, use it as the next teacher and train an -step student. Repeat until the desired budget is reached. Each stage only has to learn a factor-of-two compression, but every stage costs training and can transfer approximation errors to the next one. This is a curriculum over discretizations rather than a new forward corruption process.
Results
Table 2 reports unconditional CIFAR-10 FID [25], [22]: 9.12 at one model evaluation, 4.51 at two, 3.00 at four and 2.57 at eight. The inference computation falls, while the training stages supply the information that a coarse solver would otherwise miss. These are operating points of the progressively distilled family, not one model evaluated with an arbitrary external schedule. The four-step result preserves much more of the original model’s sample quality than an unmodified sampler at a similarly small budget.
The paper also studies larger-resolution generation, but its CIFAR result should not be read as proof that one-step text-to-image generation has the same trade-off. The task, architecture, conditioning and evaluation distribution differ. Later papers may report different progressive-distillation baselines because they reimplement or retune the method; attach a baseline to the paper that measured it.
Ablations
Table 1 compares parameterizations before distillation on unconditional CIFAR-10. With clean-image prediction and truncated-SNR weighting, its deterministic sampler gives FID 2.51; noise prediction with the ordinary SNR weighting gives 2.91. The noise-prediction/truncated-SNR combination diverges. These results support stable output/weighting choices, not a claim that 2.51 is the one-step distilled score.
The evaluation-count sweep above is a second important study: fewer stages of inference progressively worsen sample quality, so compression has a measurable limit. The paper also compares numerical integrators and studies prediction behavior across noise levels. It does not establish that repeatedly halving indefinitely can preserve an exact teacher distribution with a fixed finite network.
Conclusion
Learning a finite teacher transition is more effective than asking an unchanged denoiser to take a very large step. The target is analytically constructed by inverting the student’s deterministic update, and progressive halving provides a practical training curriculum.
Problems
The procedure requires a teacher, several training stages and compatible schedules. It preserves a selected deterministic coupling, including that coupling’s defects. Very small budgets still lose quality, and changes to guidance or conditioning after training need care. Consistency models supply a different endpoint-learning objective; distribution matching later relaxes the demand to reproduce each teacher trajectory [8], [16].
References
- Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, B. Poole, “Score-Based Generative Modeling through Stochastic Differential Equations,” ICLR, 2021. paper
- Y. Lipman, M. Havasi, P. Holderrieth, et al., “Flow Matching Guide and Code,” arXiv tutorial, 2024. paper
- T. Karras, M. Aittala, T. Aila, S. Laine, “Elucidating the Design Space of Diffusion-Based Generative Models,” NeurIPS, 2022. source
- X. Fan, Z. Wu, H. Wu, L. Cao, “A Survey on Pre-Trained Diffusion Model Distillations,” arXiv preprint, version 2, 2025. paper
- J. Song, C. Meng, S. Ermon, “Denoising Diffusion Implicit Models,” ICLR, 2021. paper
- C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li, J. Zhu, “DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps,” NeurIPS, 2022. paper
- T. Salimans, J. Ho, “Progressive Distillation for Fast Sampling of Diffusion Models,” ICLR, 2022. paper
- Y. Song, P. Dhariwal, M. Chen, I. Sutskever, “Consistency Models,” ICML, 2023. paper
- Y. Song, P. Dhariwal, “Improved Techniques for Training Consistency Models,” ICLR, 2024. paper
- S. Luo, Y. Tan, L. Huang, J. Li, H. Zhao, “Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference,” arXiv preprint, 2023. paper
- R. Rombach, A. Blattmann, D. Lorenz, P. Esser, B. Ommer, “High-Resolution Image Synthesis with Latent Diffusion Models,” CVPR, 2022. source
- X. Liu, C. Gong, Q. Liu, “Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow,” ICLR, 2023. paper
- X. Liu, X. Zhang, J. Ma, J. Peng, Q. Liu, “InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation,” ICLR, 2024. paper
- A. Sauer, D. Lorenz, A. Blattmann, R. Rombach, “Adversarial Diffusion Distillation,” ECCV, 2024. paper
- A. Sauer, F. Boesel, T. Dockhorn, A. Blattmann, P. Esser, R. Rombach, “Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion Distillation,” SIGGRAPH Asia Conference Papers, 2024. paper · published version
- T. Yin, M. Gharbi, R. Zhang, E. Shechtman, F. Durand, W. T. Freeman, T. Park, “One-step Diffusion with Distribution Matching Distillation,” CVPR, 2024. paper
- T. H. Nguyen, A. Tran, “SwiftBrush: One-Step Text-to-Image Diffusion Model with Variational Score Distillation,” CVPR, 2024. paper
- M. Zhou, H. Zheng, Z. Wang, M. Yin, H. Huang, “Score identity Distillation: Exponentially Fast Distillation of Pretrained Diffusion Models for One-Step Generation,” ICML, 2024. paper
- T. Yin, M. Gharbi, T. Park, R. Zhang, E. Shechtman, F. Durand, W. T. Freeman, “Improved Distribution Matching Distillation for Fast Image Synthesis,” NeurIPS, 2024. paper
- K. Frans, D. Hafner, S. Levine, P. Abbeel, “One Step Diffusion via Shortcut Models,” ICLR, 2025. paper
- Z. Geng, M. Deng, X. Bai, J. Z. Kolter, K. He, “Mean Flows for One-step Generative Modeling,” NeurIPS, 2025. paper
- M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, S. Hochreiter, “GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium,” NeurIPS, 2017. source
- A. Radford, J. W. Kim, C. Hallacy, et al., “Learning Transferable Visual Models From Natural Language Supervision,” ICML, 2021. paper
- D. Ghosh, H. Hajishirzi, L. Schmidt, “GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment,” NeurIPS Datasets and Benchmarks, 2023. paper
- A. Krizhevsky, V. Nair and G. Hinton, “CIFAR-10 and CIFAR-100 datasets,” official dataset page; companion technical report: A. Krizhevsky, “Learning Multiple Layers of Features from Tiny Images,” 2009. dataset and report
- J. Deng, W. Dong, R. Socher, L. Li, Kai Li, Li Fei-Fei, “ImageNet: A large-scale hierarchical image database,” CVPR, 2009. doi
- T. Lin, M. Maire, S. Belongie, et al., “Microsoft COCO: Common Objects in Context,” ECCV, 2014. paper
- L. Yang, Z. Zhang, Y. Song, et al., “Diffusion Models: A Comprehensive Survey of Methods and Applications,” ACM Computing Surveys, 2023. paper
- Z. Yu, J. Yuan, H. Yang, et al., “DMAD: Distribution Matching as Adversarial Distillation for Fast Visual Generation,” arXiv preprint, 2026. paper
- R. Zhang, P. Isola, A. A. Efros, E. Shechtman and O. Wang, “The Unreasonable Effectiveness of Deep Features as a Perceptual Metric,” in Proc. IEEE/CVF CVPR, 2018, pp. 586–595. doi · arXiv
- C. Schuhmann, R. Beaumont, R. Vencu, et al., “LAION-5B: An open large-scale dataset for training next generation image-text models,” NeurIPS Datasets and Benchmarks, 2022. paper