完整搜尋頁面 →

MeanFlow:學習整段時間的平均速度

選讀・ML 08・擴散模型與流匹配 的論文導讀(第 3/6 篇)。引用編號對應本頁底部的參考文獻。

導覽:本頁與本系列: 導覽:本頁與本系列

問題與論點

為何對 flow-matching 模型直接採一次大步長,通常會失敗?瞬時速度描述局部切線,不是整條彎曲軌跡的位移。Geng 等人改為學習區間平均速度 [13]。先讀 Flow Matching 導讀與正文的一步 Euler 反例。貢獻是改變學習的量,不只是替同一網路換較快求解器。

方法與圖解

瞬時速度沿軌跡切線,平均速度則對齊指定區間的位移,因此速度場同時取決於區間兩端時間。
來源:Geng et al., NeurIPS 2025 proceedings. 圖 3,PDF 第 4 頁。瞬時速度沿軌跡切線,平均速度則對齊指定區間的位移,因此速度場同時取決於區間兩端時間。 [13]

採論文時間慣例:zt=(1−t)x+tϵz_t=(1-t)x+t\epsilon,其中 xx 為乾淨資料,ϵ\epsilon 為先驗雜訊。零端是資料,一端是雜訊,與前一篇 flow-matching 導讀方向相反;生成往較小時間走。對邊際 ODE 軌跡,

u(zt,r,t)=1t−r∫rtv(zs,s) ds,u=v−(t−r)dudt,zr=zt−(t−r)u(zt,r,t).u(z_t,r,t)=\frac{1}{t-r}\int_r^t v(z_s,s)\,ds,\qquad u=v-(t-r)\frac{du}{dt},\qquad z_r=z_t-(t-r)u(z_t,r,t).

其中 r<tr\lt t 為區間端點,vv 為瞬時邊際速度,uu 為平均速度,zsz_s 是同一軌跡上的點。r=tr=t 邊界採極限 u=vu=v。圖 3 區分切線與位移方向,兩者不必相同。

全導數為 du/dt=∂tu+Jzu vdu/dt=\partial_tu+J_zu\,v;JzuJ_zu 是狀態 Jacobian,rr 保持固定。Jacobian-vector product 可計算此量,不必建立完整矩陣。原始配方以抽樣條件速度 ϵ−x\epsilon-x 取代邊際速度,在導數中使用當前網路,並阻斷所得迴歸目標的梯度。這些是訓練選擇,不是直接取得未知的精確區間平均。

證據

表 1 用 B/4 骨幹、80 個 epoch,在 256×256 ImageNet 的 1-NFE 生成測試區間機制。只抽零長區間會退化成一般 flow matching,一次呼叫取樣時 FID 為 328.91;25% 非零區間比例則為 61.06。這是消融設定,不是大型模型的摘要結果。表 2 另外報告 MeanFlow-XL/2 訓練 240 個 epoch、1 NFE 時的 FID 3.43。

設計分析

恆等式連結不同長度區間;正確時間導數與空間切線同樣重要,因此論文刻意改錯 JVP 切線來驗證。引導被納入學得速度場,推論才能維持一次網路評估;若之後再套一般雙次 classifier-free guidance,NFE 就改變了。此外,ImageNet 生成器是在預訓練 VAE 的潛在空間從頭訓練,「從頭」不會抹去 tokenizer 已有的訓練。

閱讀檢查點

先對等速直線軌跡比較瞬時與平均速度,再對彎曲軌跡重做推理。說明為何 z0=z1−uθ(z1,0,1)z_0=z_1-u_\theta(z_1,0,1) 是一次學得區間預測,而不是原始瞬時場的精確 Euler 大步。

限制與下一步

網路只近似恆等式,目標也取決於當前預測。一步生成移除重複積分,卻沒有移除訓練難度、tokenizer 成本或分布誤差。接著讀 Improved MeanFlow,檢查預測 JVP 方向如何改變迴歸輸入,以及彈性引導為何需要額外條件。

接著閱讀: Improved MeanFlow:預測輸入與彈性引導

參考文獻

  1. L. Yang, Z. Zhang, Y. Song, S. Hong, R. Xu, Y. Zhao, W. Zhang, B. Cui and M.-H. Yang, “Diffusion Models: A Comprehensive Survey of Methods and Applications,” ACM Computing Surveys, 2023. arXiv
  2. Y. Lipman, M. Havasi, P. Holderrieth, N. Shaul, M. Le, B. Karrer, R. T. Q. Chen, D. Lopez-Paz, H. Ben-Hamu and I. Gat, “Flow Matching Guide and Code,” arXiv:2412.06264, 2024. arXiv
  3. J. Ho, A. Jain and P. Abbeel, “Denoising Diffusion Probabilistic Models,” NeurIPS, 2020. arXiv
  4. Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon and B. Poole, “Score-Based Generative Modeling through Stochastic Differential Equations,” ICLR, 2021. arXiv
  5. J. Song, C. Meng and S. Ermon, “Denoising Diffusion Implicit Models,” ICLR, 2021. arXiv
  6. J. Ho and T. Salimans, “Classifier-Free Diffusion Guidance,” NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications; arXiv:2207.12598, 2022. arXiv
  7. R. Rombach, A. Blattmann, D. Lorenz, P. Esser and B. Ommer, “High-Resolution Image Synthesis with Latent Diffusion Models,” CVPR, 2022. arXiv
  8. W. Peebles and S. Xie, “Scalable Diffusion Models with Transformers,” ICCV, 2023. arXiv
  9. P. Esser, S. Kulal, A. Blattmann et al., “Scaling Rectified Flow Transformers for High-Resolution Image Synthesis,” ICML, PMLR 235, 2024. PMLR
  10. Y. Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel and M. Le, “Flow Matching for Generative Modeling,” ICLR, 2023. arXiv
  11. X. Liu, C. Gong and Q. Liu, “Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow,” ICLR, 2023. arXiv
  12. Y. Song, P. Dhariwal, M. Chen and I. Sutskever, “Consistency Models,” ICML, 2023. arXiv
  13. Z. Geng, M. Deng, X. Bai, J. Z. Kolter and K. He, “Mean Flows for One-step Generative Modeling,” NeurIPS, 2025. arXiv
  14. S. J. D. Prince, Understanding Deep Learning, MIT Press, 2023. book site
  15. J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan and S. Ganguli, “Deep Unsupervised Learning using Nonequilibrium Thermodynamics,” ICML, PMLR 37, 2015. PMLR
  16. P. Vincent, “A Connection Between Score Matching and Denoising Autoencoders,” Neural Computation, vol. 23, no. 7, pp. 1661–1674, 2011. doi
  17. Y. Song and S. Ermon, “Generative Modeling by Estimating Gradients of the Data Distribution,” NeurIPS, 2019. arXiv
  18. P. Dhariwal and A. Nichol, “Diffusion Models Beat GANs on Image Synthesis,” NeurIPS, 2021. NeurIPS
  19. A. Q. Nichol and P. Dhariwal, “Improved Denoising Diffusion Probabilistic Models,” ICML, PMLR 139, 2021. PMLR
  20. T. Karras, M. Aittala, T. Aila and S. Laine, “Elucidating the Design Space of Diffusion-Based Generative Models,” NeurIPS, 2022. arXiv
  21. C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li and J. Zhu, “DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps,” NeurIPS, 2022. arXiv
  22. T. Salimans and J. Ho, “Progressive Distillation for Fast Sampling of Diffusion Models,” ICLR, 2022. arXiv
  23. M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler and S. Hochreiter, “GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium,” NeurIPS (NIPS), 2017. arXiv
  24. T. Kynkäänniemi, T. Karras, S. Laine, J. Lehtinen and T. Aila, “Improved Precision and Recall Metric for Assessing Generative Models,” NeurIPS, 2019. arXiv
  25. G. Stein, J. C. Cresswell, R. Hosseinzadeh et al., “Exposing Flaws of Generative Model Evaluation Metrics and Their Unfair Treatment of Diffusion Models,” NeurIPS, 2023. arXiv
  26. S. Jayasumana, S. Ramalingam, A. Veit, D. Glasner, A. Chakrabarti and S. Kumar, “Rethinking FID: Towards a Better Evaluation Metric for Image Generation,” CVPR, 2024. CVF
  27. F.-A. Croitoru, V. Hondru, R. T. Ionescu and M. Shah, “Diffusion Models in Vision: A Survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023. doi
  28. H. Cao, C. Tan, Z. Gao, Y. Xu, G. Chen, P.-A. Heng and S. Z. Li, “A Survey on Generative Diffusion Models,” IEEE Transactions on Knowledge and Data Engineering, vol. 36, no. 7, pp. 2814–2830, 2024. doi
  29. C.-H. Lai, Y. Song, D. Kim, Y. Mitsufuji and S. Ermon, “The Principles of Diffusion Models,” monograph, arXiv:2510.21890, 2025. arXiv
  30. H. Shen, J. Zhang, B. Xiong et al., “Efficient Diffusion Models: A Survey,” Transactions on Machine Learning Research, 2025. arXiv
  31. A. Melnik, M. Ljubljanac, C. Lu, Q. Yan, W. Ren and H. Ritter, “Video Diffusion Models: A Survey,” Transactions on Machine Learning Research, 2024. arXiv
  32. C. Lu and Y. Song, “Simplifying, Stabilizing and Scaling Continuous-Time Consistency Models,” ICLR, 2025. arXiv
  33. A. Sauer, D. Lorenz, A. Blattmann and R. Rombach, “Adversarial Diffusion Distillation,” ECCV, 2024. arXiv
  34. M. S. Albergo and E. Vanden-Eijnden, “Building Normalizing Flows with Stochastic Interpolants,” ICLR, 2023. arXiv
  35. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li and L. Fei-Fei, “ImageNet: A Large-Scale Hierarchical Image Database,” CVPR, 2009, pp. 248–255. doi
  36. C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens and Z. Wojna, “Rethinking the Inception Architecture for Computer Vision,” CVPR, 2016, pp. 2818–2826. arXiv
  37. O. Ronneberger, P. Fischer and T. Brox, “U-Net: Convolutional Networks for Biomedical Image Segmentation,” MICCAI, LNCS 9351, pp. 234–241, 2015. arXiv
  38. A. Radford, J. W. Kim, C. Hallacy et al., “Learning Transferable Visual Models From Natural Language Supervision,” ICML, 2021. arXiv
  39. M. Oquab, T. Darcet, T. Moutakanni et al., “DINOv2: Learning Robust Visual Features without Supervision,” Transactions on Machine Learning Research, 2024. arXiv
  40. B. Efron, “Tweedie’s Formula and Selection Bias,” Journal of the American Statistical Association, vol. 106, no. 496, pp. 1602–1614, 2011. doi
  41. A. Dosovitskiy, L. Beyer, A. Kolesnikov et al., “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale,” ICLR, 2021. arXiv
  42. T. Li and K. He, “Back to Basics: Let Denoising Generative Models Denoise,” CVPR, 2026 (preprint 2025). paper, publication record, official JiT code.
  43. B. Zheng, N. Ma, S. Tong and S. Xie, “Diffusion Transformers with Representation Autoencoders,” ICLR, 2026 (preprint 2025). proceedings, manuscript.
  44. S. Tong, B. Zheng, Z. Wang, B. Tang, N. Ma, E. Brown, J. Yang, R. Fergus, Y. LeCun and S. Xie, “Scaling Text-to-Image Diffusion Transformers with Representation Autoencoders,” CVPR Findings Track, 2026. manuscript, author publication record.
  45. D. Podell, Z. English, K. Lacey, A. Blattmann, T. Dockhorn, J. Müller, J. Penna and R. Rombach, “SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis,” ICLR, 2024. proceedings, official model card.
  46. Black Forest Labs, “FLUX.1 [dev],” official pretrained-model card, 2024. model card.
  47. W. Kong, Q. Tian, Z. Zhang et al., “HunyuanVideo: A Systematic Framework For Large Video Generative Models,” technical report, arXiv:2412.03603, 2024. report, official release.
  48. Team Wan, A. Wang, B. Ai, B. Wen et al., “Wan: Open and Advanced Large-Scale Video Generative Models,” technical report, arXiv:2503.20314, 2025. report, official Wan2.1 release.
  49. Z. Geng, Y. Lu, Z. Wu, E. Shechtman, J. Z. Kolter and K. He, “Improved Mean Flows: On the Challenges of Fastforward Generative Models,” CVPR, pp. 30467–30476, 2026. CVF
  50. Y. Lu, S. Lu, Q. Sun, H. Zhao, Z. Jiang, X. Wang, T. Li, Z. Geng and K. He, “One-step Latent-free Image Generation with Pixel Mean Flows,” ICML, PMLR 306, pp. 81265–81277, 2026. PMLR