Progressive Distillation:教師兩步壓成學生一步(ICLR 2022)
選讀・DLCV 12B・單步生成 的論文導讀(第 1/7 篇)。引用編號對應本頁底部的參考文獻。
導覽:本頁與本系列: 導覽:本頁與本系列
挑戰
準確的擴散取樣器常需多次網路評估。Salimans 與 Ho [7] 不只減少原求解器步數,而讓學生學會教師確定性兩步的效果,把積分工作攤到訓練。
含雜訊觀測可能對應多張乾淨圖,因此一般平方誤差去雜訊會學平均;固定起點與確定性教師後,教師終點則唯一。迴歸該終點可避免平均無關候選,但會繼承教師選定的映射。
方法

圖片來源: [7], arXiv:2202.00512v2, PDF 第 2 頁. 原始 PDF.
教師與排程。 使用預訓練擴散模型及確定性 DDIM [5]。,其中 為資料, 為標準高斯, 為排程。學生沿用架構及教師初始化,時間從學生網格抽樣;終端訊雜比為零,符合純雜訊生成。
教師兩步。 從 先走到中點,再到 ,產生 。這只用來建構訓練目標;部署不執行教師。狀態與目標維持原影像/潛在張量形狀,不必把整串中間狀態輸入學生。
反解學生更新。 DDIM 單步對乾淨預測為仿射,所需目標為:
是共同起點, 是教師終點, 是較乾淨時間。預測 即可透過單步 DDIM 重現教師兩步。學生最小化排程加權平方誤差,不是直接模仿原乾淨樣本或教師的第一次預測。
輸出參數化。 純雜訊邊界下,從雜訊預測換算乾淨圖需除以 ,容易奇異。原文比較乾淨、乾淨/雜訊組合及速度預測,搭配訊雜比權重。變異數保留排程的 是乾淨/雜訊座標旋轉;參數化與權重共同決定不同雜訊程度的影響。
逐步壓縮。 訓練 步學生後讓它當教師,再訓練 步模型。每階段只壓縮兩倍,但都要付出訓練成本並可能傳遞誤差。這是離散化的學習課程,沒有改變前向加雜訊程序。
結果
表 2 的無條件 CIFAR-10 FID [25]、[22]:一次評估 9.12,兩次 4.51,四次 3.00,八次 2.57。這是逐階段蒸餾家族的操作點,不能解讀成任意替單一模型換排程。少步需要的資訊來自訓練,仍有品質代價。
原文也研究較大解析度,但小影像結果不能直接證明單步文字生成的取捨。任務、條件與架構皆不同;後續論文重新實作的基準,也應歸於實際測量它的設定。
消融實驗
表 1 比較的是蒸餾前參數化:乾淨預測加 truncated-SNR 的 FID 為 2.51,雜訊預測搭配一般 SNR 為 2.91;雜訊預測加 truncated-SNR 發散。2.51 並非單步蒸餾成績。
評估次數掃描也顯示壓縮極限:減少推論會逐漸犧牲品質。求解器及雜訊程度分析不代表固定容量網路可無限減半仍精確保留教師分布。
結論
學習有限教師轉移,比要求原去雜訊器直接跨大步更有效。解析反解更新式提供目標,逐次減半提供可行的訓練課程。
限制與問題
方法仍需要教師、多階段訓練與相容排程,也保留特定確定性耦合的缺陷。極少步會降質,改動引導須謹慎。一致性 [8] 改學端點關係,分布匹配 [16] 則放寬逐軌跡對應。
參考文獻
- Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, B. Poole, “Score-Based Generative Modeling through Stochastic Differential Equations,” ICLR, 2021. paper
- Y. Lipman, M. Havasi, P. Holderrieth, et al., “Flow Matching Guide and Code,” arXiv tutorial, 2024. paper
- T. Karras, M. Aittala, T. Aila, S. Laine, “Elucidating the Design Space of Diffusion-Based Generative Models,” NeurIPS, 2022. source
- X. Fan, Z. Wu, H. Wu, L. Cao, “A Survey on Pre-Trained Diffusion Model Distillations,” arXiv preprint, version 2, 2025. paper
- J. Song, C. Meng, S. Ermon, “Denoising Diffusion Implicit Models,” ICLR, 2021. paper
- C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li, J. Zhu, “DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps,” NeurIPS, 2022. paper
- T. Salimans, J. Ho, “Progressive Distillation for Fast Sampling of Diffusion Models,” ICLR, 2022. paper
- Y. Song, P. Dhariwal, M. Chen, I. Sutskever, “Consistency Models,” ICML, 2023. paper
- Y. Song, P. Dhariwal, “Improved Techniques for Training Consistency Models,” ICLR, 2024. paper
- S. Luo, Y. Tan, L. Huang, J. Li, H. Zhao, “Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference,” arXiv preprint, 2023. paper
- R. Rombach, A. Blattmann, D. Lorenz, P. Esser, B. Ommer, “High-Resolution Image Synthesis with Latent Diffusion Models,” CVPR, 2022. source
- X. Liu, C. Gong, Q. Liu, “Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow,” ICLR, 2023. paper
- X. Liu, X. Zhang, J. Ma, J. Peng, Q. Liu, “InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation,” ICLR, 2024. paper
- A. Sauer, D. Lorenz, A. Blattmann, R. Rombach, “Adversarial Diffusion Distillation,” ECCV, 2024. paper
- A. Sauer, F. Boesel, T. Dockhorn, A. Blattmann, P. Esser, R. Rombach, “Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion Distillation,” SIGGRAPH Asia Conference Papers, 2024. paper · published version
- T. Yin, M. Gharbi, R. Zhang, E. Shechtman, F. Durand, W. T. Freeman, T. Park, “One-step Diffusion with Distribution Matching Distillation,” CVPR, 2024. paper
- T. H. Nguyen, A. Tran, “SwiftBrush: One-Step Text-to-Image Diffusion Model with Variational Score Distillation,” CVPR, 2024. paper
- M. Zhou, H. Zheng, Z. Wang, M. Yin, H. Huang, “Score identity Distillation: Exponentially Fast Distillation of Pretrained Diffusion Models for One-Step Generation,” ICML, 2024. paper
- T. Yin, M. Gharbi, T. Park, R. Zhang, E. Shechtman, F. Durand, W. T. Freeman, “Improved Distribution Matching Distillation for Fast Image Synthesis,” NeurIPS, 2024. paper
- K. Frans, D. Hafner, S. Levine, P. Abbeel, “One Step Diffusion via Shortcut Models,” ICLR, 2025. paper
- Z. Geng, M. Deng, X. Bai, J. Z. Kolter, K. He, “Mean Flows for One-step Generative Modeling,” NeurIPS, 2025. paper
- M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, S. Hochreiter, “GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium,” NeurIPS, 2017. source
- A. Radford, J. W. Kim, C. Hallacy, et al., “Learning Transferable Visual Models From Natural Language Supervision,” ICML, 2021. paper
- D. Ghosh, H. Hajishirzi, L. Schmidt, “GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment,” NeurIPS Datasets and Benchmarks, 2023. paper
- A. Krizhevsky, V. Nair and G. Hinton, “CIFAR-10 and CIFAR-100 datasets,” official dataset page; companion technical report: A. Krizhevsky, “Learning Multiple Layers of Features from Tiny Images,” 2009. dataset and report
- J. Deng, W. Dong, R. Socher, L. Li, Kai Li, Li Fei-Fei, “ImageNet: A large-scale hierarchical image database,” CVPR, 2009. doi
- T. Lin, M. Maire, S. Belongie, et al., “Microsoft COCO: Common Objects in Context,” ECCV, 2014. paper
- L. Yang, Z. Zhang, Y. Song, et al., “Diffusion Models: A Comprehensive Survey of Methods and Applications,” ACM Computing Surveys, 2023. paper
- Z. Yu, J. Yuan, H. Yang, et al., “DMAD: Distribution Matching as Adversarial Distillation for Fast Visual Generation,” arXiv preprint, 2026. paper
- R. Zhang, P. Isola, A. A. Efros, E. Shechtman and O. Wang, “The Unreasonable Effectiveness of Deep Features as a Perceptual Metric,” in Proc. IEEE/CVF CVPR, 2018, pp. 586–595. doi · arXiv
- C. Schuhmann, R. Beaumont, R. Vencu, et al., “LAION-5B: An open large-scale dataset for training next generation image-text models,” NeurIPS Datasets and Benchmarks, 2022. paper