DMD2:追蹤學生,加入真實資料(NeurIPS 2024)
選讀・DLCV 12B・單步生成 的論文導讀(第 7/7 篇)。引用編號對應本頁底部的參考文獻。
導覽:本頁與本系列: 導覽:本頁與本系列
挑戰
DMD2 [19] 問:移除昂貴的教師端點資料後,為什麼分布匹配會不穩?關鍵之一是輔助分數未能準確追蹤變動學生分布。
另一問題是固定教師的系統偏差;只匹配它無法取得真實影像的修正證據。多步學生還有訓練的前向加雜訊狀態與實際推論中間狀態不匹配。
方法

圖片來源: [19], arXiv:2405.14867v2, PDF 第 6 頁. 原始 PDF.
移除端點迴歸。 保留固定目標分數及可訓練學生分數,不再以種子配對預計算影像。節省離線多步目標,但更依賴分數差準確度。
雙時間尺度。 原設定每次生成器更新前,輔助分數更新 5 次,讓密度估計先跟上目前分布。這增加訓練成本,不增加推論步,也不讓分數自動精確。
真實影像判別器。 在輔助去雜訊器瓶頸特徵加分類頭,以不同雜訊程度的真實/生成圖做對抗學習。共用表示避免獨立大型判別網路,提供超出教師密度的監督。生成器接收分布及對抗梯度。
多步學生。 訓練以學生自己的先前取樣步產生下一個輸入,稱為 backward simulation,減少前向加雜訊與部署狀態的不匹配。它模擬學生軌跡,不是恢復已移除的教師端點迴歸資料。
引導與表示。 目標分數條件須符合生成提示,可把教師引導蒸餾進學生。潛在模型仍要解碼;教師、輔助去雜訊器與判別頭皆可在推論移除。
三項設計分工不同:額外分數更新改善追蹤;真實對抗資料修正教師目標偏差;狀態模擬改變多步訓練輸入。單靠追蹤不會消除教師偏差,單靠對抗也不等於擴散分布目標。
結果
ImageNet-64 一步 FID 1.51,延長訓練為 1.28。教師 ODE 取樣 2.32,SDE 1.36;「超越教師」必須說明比較哪個取樣器,而非宣稱所有操作點及指標皆更好。
表 2 的 SDXL、COCO-2014 10,000 提示:一步 FID 19.01、CLIP 0.336;四步 19.32/0.332。四步 patch FID 則從 26.98 改善為 20.86,顯示指標偏好的步數不同。10k 與 30k 規則也不同,不能互換。
消融實驗
ImageNet 消融:原 DMD 2.62,刪迴歸 3.48,加入不同更新頻率 2.61,再加入真實 GAN 1.51。GAN-only 為 2.56,加更新頻率為 2.52;完整組合優於這些單獨設計。
四步 SDXL 移除 GAN 為 FID/patch FID 26.90/27.66,完整 19.32/20.86。移除分布匹配的整圖 FID 反而 13.77,但 patch FID 27.96、CLIP 0.307,差於完整 0.332。移除狀態模擬為 20.66/24.21。只看一個指標會得出錯誤排序。
結論
分布蒸餾需要準確追蹤學生。平衡更新可取代原始端點正規化,真實資料與學生狀態則分別處理目標和輸入誤差。
限制與問題
訓練仍需大型預訓練模型、輔助更新與真實資料。基準改善不保證罕見模態或組合正確性;狀態模擬也增加訓練工作。部署變快不代表訓練廉價或完全沒有教師偏差;後續對抗分布方法 [29] 探討其他免追蹤途徑。
參考文獻
- Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, B. Poole, “Score-Based Generative Modeling through Stochastic Differential Equations,” ICLR, 2021. paper
- Y. Lipman, M. Havasi, P. Holderrieth, et al., “Flow Matching Guide and Code,” arXiv tutorial, 2024. paper
- T. Karras, M. Aittala, T. Aila, S. Laine, “Elucidating the Design Space of Diffusion-Based Generative Models,” NeurIPS, 2022. source
- X. Fan, Z. Wu, H. Wu, L. Cao, “A Survey on Pre-Trained Diffusion Model Distillations,” arXiv preprint, version 2, 2025. paper
- J. Song, C. Meng, S. Ermon, “Denoising Diffusion Implicit Models,” ICLR, 2021. paper
- C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li, J. Zhu, “DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps,” NeurIPS, 2022. paper
- T. Salimans, J. Ho, “Progressive Distillation for Fast Sampling of Diffusion Models,” ICLR, 2022. paper
- Y. Song, P. Dhariwal, M. Chen, I. Sutskever, “Consistency Models,” ICML, 2023. paper
- Y. Song, P. Dhariwal, “Improved Techniques for Training Consistency Models,” ICLR, 2024. paper
- S. Luo, Y. Tan, L. Huang, J. Li, H. Zhao, “Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference,” arXiv preprint, 2023. paper
- R. Rombach, A. Blattmann, D. Lorenz, P. Esser, B. Ommer, “High-Resolution Image Synthesis with Latent Diffusion Models,” CVPR, 2022. source
- X. Liu, C. Gong, Q. Liu, “Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow,” ICLR, 2023. paper
- X. Liu, X. Zhang, J. Ma, J. Peng, Q. Liu, “InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation,” ICLR, 2024. paper
- A. Sauer, D. Lorenz, A. Blattmann, R. Rombach, “Adversarial Diffusion Distillation,” ECCV, 2024. paper
- A. Sauer, F. Boesel, T. Dockhorn, A. Blattmann, P. Esser, R. Rombach, “Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion Distillation,” SIGGRAPH Asia Conference Papers, 2024. paper · published version
- T. Yin, M. Gharbi, R. Zhang, E. Shechtman, F. Durand, W. T. Freeman, T. Park, “One-step Diffusion with Distribution Matching Distillation,” CVPR, 2024. paper
- T. H. Nguyen, A. Tran, “SwiftBrush: One-Step Text-to-Image Diffusion Model with Variational Score Distillation,” CVPR, 2024. paper
- M. Zhou, H. Zheng, Z. Wang, M. Yin, H. Huang, “Score identity Distillation: Exponentially Fast Distillation of Pretrained Diffusion Models for One-Step Generation,” ICML, 2024. paper
- T. Yin, M. Gharbi, T. Park, R. Zhang, E. Shechtman, F. Durand, W. T. Freeman, “Improved Distribution Matching Distillation for Fast Image Synthesis,” NeurIPS, 2024. paper
- K. Frans, D. Hafner, S. Levine, P. Abbeel, “One Step Diffusion via Shortcut Models,” ICLR, 2025. paper
- Z. Geng, M. Deng, X. Bai, J. Z. Kolter, K. He, “Mean Flows for One-step Generative Modeling,” NeurIPS, 2025. paper
- M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, S. Hochreiter, “GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium,” NeurIPS, 2017. source
- A. Radford, J. W. Kim, C. Hallacy, et al., “Learning Transferable Visual Models From Natural Language Supervision,” ICML, 2021. paper
- D. Ghosh, H. Hajishirzi, L. Schmidt, “GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment,” NeurIPS Datasets and Benchmarks, 2023. paper
- A. Krizhevsky, V. Nair and G. Hinton, “CIFAR-10 and CIFAR-100 datasets,” official dataset page; companion technical report: A. Krizhevsky, “Learning Multiple Layers of Features from Tiny Images,” 2009. dataset and report
- J. Deng, W. Dong, R. Socher, L. Li, Kai Li, Li Fei-Fei, “ImageNet: A large-scale hierarchical image database,” CVPR, 2009. doi
- T. Lin, M. Maire, S. Belongie, et al., “Microsoft COCO: Common Objects in Context,” ECCV, 2014. paper
- L. Yang, Z. Zhang, Y. Song, et al., “Diffusion Models: A Comprehensive Survey of Methods and Applications,” ACM Computing Surveys, 2023. paper
- Z. Yu, J. Yuan, H. Yang, et al., “DMAD: Distribution Matching as Adversarial Distillation for Fast Visual Generation,” arXiv preprint, 2026. paper
- R. Zhang, P. Isola, A. A. Efros, E. Shechtman and O. Wang, “The Unreasonable Effectiveness of Deep Features as a Perceptual Metric,” in Proc. IEEE/CVF CVPR, 2018, pp. 586–595. doi · arXiv
- C. Schuhmann, R. Beaumont, R. Vencu, et al., “LAION-5B: An open large-scale dataset for training next generation image-text models,” NeurIPS Datasets and Benchmarks, 2022. paper