完整搜尋頁面 →

JiT:在像素空間直接預測乾淨影像

選讀・ML 08・擴散模型與流匹配 的論文導讀(第 5/6 篇)。引用編號對應本頁底部的參考文獻。

導覽:本頁與本系列: 導覽:本頁與本系列

問題與論點

為何大區塊像素 Transformer 在直接預測乾淨影像時可行,改成直接預測雜訊或速度卻可能失敗?Li 與 He 主張,自然乾淨影像的結構維度遠低於周圍帶雜訊空間 [42]。Just image Transformers,簡稱 JiT,研究預測目標與網路容量的互動。先讀 ViT 區塊維度,以及正文中直接輸出與損失空間的區分。

方法與圖解

像素區塊經一般 Transformer 與線性輸出投影,形成乾淨影像預測;架構不需要潛在 tokenizer,但仍使用取樣軌跡。
來源:Li and He, CVPR 2026 proceedings. 圖 3,PDF 第 5 頁。像素區塊經一般 Transformer 與線性輸出投影,形成乾淨影像預測;架構不需要潛在 tokenizer,但仍使用取樣軌跡。 [42]

圖 3 包含像素區塊、線性嵌入、Transformer 區塊及影像區塊的線性預測;時間與類別作為條件。生成器沒有另一組潛在編碼器/解碼器。然而,在每個時間預測乾淨影像,不等於學到完整的一步生成映射。

為明示時間方向差異,以 ss 表示 JiT 的時間:零端是雜訊,一端是資料。則

ys=(1−s)ϵ+sx,v^θ=x^θ(ys,s)−ys1−s,∥v^θ−(x−ϵ)∥2=∥x^θ−x∥2(1−s)2.y_s=(1-s)\epsilon+s x,\qquad \widehat v_\theta=\frac{\widehat x_\theta(y_s,s)-y_s}{1-s},\qquad \|\widehat v_\theta-(x-\epsilon)\|^2=\frac{\|\widehat x_\theta-x\|^2}{(1-s)^2}.

其中 xx 為乾淨資料,ϵ\epsilon 為先驗雜訊,ysy_s 為插值輸入,x^θ\widehat x_\theta 為網路直接輸出,v^θ\widehat v_\theta 為導出的瞬時速度。式子在 s<1s\lt1、尚未加入數值保護時成立。換成 MeanFlow 慣例需代入 t=1−st=1-s,並反轉速度正負號。即使網路預測影像,求解器仍使用速度;速度損失等於時間加權影像誤差,不是未加權影像 MSE。

證據

表 2 交叉比較三種直接輸出與三種損失空間。JiT-B/16 在 256×256 ImageNet、200 個 epoch 與 CFG 下,速度損失那列的影像輸出 FID 為 8.62,雜訊輸出 372.38,速度輸出 96.53。較小的 64×64、區塊寬 4 設定,沒有同樣明顯的差距。固定損失後,可看出與容量相關的預測空間效果。圖 4 另顯示,影像預測搭配瓶頸區塊嵌入仍可有效。

設計分析

每個區塊的純量維度,不等於區塊數量。同時增加影像解析度及區塊大小,可固定 token 數,卻增加每個 token 所承載的資訊。人工嵌入實驗與影像消融,在有限網路容量下支持流形解釋,但沒有證明所有影像預測器都必然勝過所有雜訊預測器。只調損失權重或雜訊抽樣,也不能重現整個架構比較。

閱讀檢查點

分別指出直接輸出、損失目標,以及 ODE 求解器消耗的量。再解釋為何從高度帶雜訊輸入得到的一次乾淨預測,可能是多種可能性的平均,而非影像分布中的清晰樣本。

限制與下一步

JiT 研究類別條件生成,仍需數值取樣,本身不是一步或自由文字生成影像模型。它不使用 tokenizer 的配方,也不能與後續所有像素生成器是否使用預訓練損失混為一談。接著讀 Pixel MeanFlow,理解它如何加入雙時間去雜訊影像場及區間目標,以支援一次呼叫像素生成。

接著閱讀: Pixel MeanFlow:透過去雜訊影像場一步生成像素

參考文獻

  1. L. Yang, Z. Zhang, Y. Song, S. Hong, R. Xu, Y. Zhao, W. Zhang, B. Cui and M.-H. Yang, “Diffusion Models: A Comprehensive Survey of Methods and Applications,” ACM Computing Surveys, 2023. arXiv
  2. Y. Lipman, M. Havasi, P. Holderrieth, N. Shaul, M. Le, B. Karrer, R. T. Q. Chen, D. Lopez-Paz, H. Ben-Hamu and I. Gat, “Flow Matching Guide and Code,” arXiv:2412.06264, 2024. arXiv
  3. J. Ho, A. Jain and P. Abbeel, “Denoising Diffusion Probabilistic Models,” NeurIPS, 2020. arXiv
  4. Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon and B. Poole, “Score-Based Generative Modeling through Stochastic Differential Equations,” ICLR, 2021. arXiv
  5. J. Song, C. Meng and S. Ermon, “Denoising Diffusion Implicit Models,” ICLR, 2021. arXiv
  6. J. Ho and T. Salimans, “Classifier-Free Diffusion Guidance,” NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications; arXiv:2207.12598, 2022. arXiv
  7. R. Rombach, A. Blattmann, D. Lorenz, P. Esser and B. Ommer, “High-Resolution Image Synthesis with Latent Diffusion Models,” CVPR, 2022. arXiv
  8. W. Peebles and S. Xie, “Scalable Diffusion Models with Transformers,” ICCV, 2023. arXiv
  9. P. Esser, S. Kulal, A. Blattmann et al., “Scaling Rectified Flow Transformers for High-Resolution Image Synthesis,” ICML, PMLR 235, 2024. PMLR
  10. Y. Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel and M. Le, “Flow Matching for Generative Modeling,” ICLR, 2023. arXiv
  11. X. Liu, C. Gong and Q. Liu, “Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow,” ICLR, 2023. arXiv
  12. Y. Song, P. Dhariwal, M. Chen and I. Sutskever, “Consistency Models,” ICML, 2023. arXiv
  13. Z. Geng, M. Deng, X. Bai, J. Z. Kolter and K. He, “Mean Flows for One-step Generative Modeling,” NeurIPS, 2025. arXiv
  14. S. J. D. Prince, Understanding Deep Learning, MIT Press, 2023. book site
  15. J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan and S. Ganguli, “Deep Unsupervised Learning using Nonequilibrium Thermodynamics,” ICML, PMLR 37, 2015. PMLR
  16. P. Vincent, “A Connection Between Score Matching and Denoising Autoencoders,” Neural Computation, vol. 23, no. 7, pp. 1661–1674, 2011. doi
  17. Y. Song and S. Ermon, “Generative Modeling by Estimating Gradients of the Data Distribution,” NeurIPS, 2019. arXiv
  18. P. Dhariwal and A. Nichol, “Diffusion Models Beat GANs on Image Synthesis,” NeurIPS, 2021. NeurIPS
  19. A. Q. Nichol and P. Dhariwal, “Improved Denoising Diffusion Probabilistic Models,” ICML, PMLR 139, 2021. PMLR
  20. T. Karras, M. Aittala, T. Aila and S. Laine, “Elucidating the Design Space of Diffusion-Based Generative Models,” NeurIPS, 2022. arXiv
  21. C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li and J. Zhu, “DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps,” NeurIPS, 2022. arXiv
  22. T. Salimans and J. Ho, “Progressive Distillation for Fast Sampling of Diffusion Models,” ICLR, 2022. arXiv
  23. M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler and S. Hochreiter, “GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium,” NeurIPS (NIPS), 2017. arXiv
  24. T. Kynkäänniemi, T. Karras, S. Laine, J. Lehtinen and T. Aila, “Improved Precision and Recall Metric for Assessing Generative Models,” NeurIPS, 2019. arXiv
  25. G. Stein, J. C. Cresswell, R. Hosseinzadeh et al., “Exposing Flaws of Generative Model Evaluation Metrics and Their Unfair Treatment of Diffusion Models,” NeurIPS, 2023. arXiv
  26. S. Jayasumana, S. Ramalingam, A. Veit, D. Glasner, A. Chakrabarti and S. Kumar, “Rethinking FID: Towards a Better Evaluation Metric for Image Generation,” CVPR, 2024. CVF
  27. F.-A. Croitoru, V. Hondru, R. T. Ionescu and M. Shah, “Diffusion Models in Vision: A Survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023. doi
  28. H. Cao, C. Tan, Z. Gao, Y. Xu, G. Chen, P.-A. Heng and S. Z. Li, “A Survey on Generative Diffusion Models,” IEEE Transactions on Knowledge and Data Engineering, vol. 36, no. 7, pp. 2814–2830, 2024. doi
  29. C.-H. Lai, Y. Song, D. Kim, Y. Mitsufuji and S. Ermon, “The Principles of Diffusion Models,” monograph, arXiv:2510.21890, 2025. arXiv
  30. H. Shen, J. Zhang, B. Xiong et al., “Efficient Diffusion Models: A Survey,” Transactions on Machine Learning Research, 2025. arXiv
  31. A. Melnik, M. Ljubljanac, C. Lu, Q. Yan, W. Ren and H. Ritter, “Video Diffusion Models: A Survey,” Transactions on Machine Learning Research, 2024. arXiv
  32. C. Lu and Y. Song, “Simplifying, Stabilizing and Scaling Continuous-Time Consistency Models,” ICLR, 2025. arXiv
  33. A. Sauer, D. Lorenz, A. Blattmann and R. Rombach, “Adversarial Diffusion Distillation,” ECCV, 2024. arXiv
  34. M. S. Albergo and E. Vanden-Eijnden, “Building Normalizing Flows with Stochastic Interpolants,” ICLR, 2023. arXiv
  35. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li and L. Fei-Fei, “ImageNet: A Large-Scale Hierarchical Image Database,” CVPR, 2009, pp. 248–255. doi
  36. C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens and Z. Wojna, “Rethinking the Inception Architecture for Computer Vision,” CVPR, 2016, pp. 2818–2826. arXiv
  37. O. Ronneberger, P. Fischer and T. Brox, “U-Net: Convolutional Networks for Biomedical Image Segmentation,” MICCAI, LNCS 9351, pp. 234–241, 2015. arXiv
  38. A. Radford, J. W. Kim, C. Hallacy et al., “Learning Transferable Visual Models From Natural Language Supervision,” ICML, 2021. arXiv
  39. M. Oquab, T. Darcet, T. Moutakanni et al., “DINOv2: Learning Robust Visual Features without Supervision,” Transactions on Machine Learning Research, 2024. arXiv
  40. B. Efron, “Tweedie’s Formula and Selection Bias,” Journal of the American Statistical Association, vol. 106, no. 496, pp. 1602–1614, 2011. doi
  41. A. Dosovitskiy, L. Beyer, A. Kolesnikov et al., “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale,” ICLR, 2021. arXiv
  42. T. Li and K. He, “Back to Basics: Let Denoising Generative Models Denoise,” CVPR, 2026 (preprint 2025). paper, publication record, official JiT code.
  43. B. Zheng, N. Ma, S. Tong and S. Xie, “Diffusion Transformers with Representation Autoencoders,” ICLR, 2026 (preprint 2025). proceedings, manuscript.
  44. S. Tong, B. Zheng, Z. Wang, B. Tang, N. Ma, E. Brown, J. Yang, R. Fergus, Y. LeCun and S. Xie, “Scaling Text-to-Image Diffusion Transformers with Representation Autoencoders,” CVPR Findings Track, 2026. manuscript, author publication record.
  45. D. Podell, Z. English, K. Lacey, A. Blattmann, T. Dockhorn, J. Müller, J. Penna and R. Rombach, “SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis,” ICLR, 2024. proceedings, official model card.
  46. Black Forest Labs, “FLUX.1 [dev],” official pretrained-model card, 2024. model card.
  47. W. Kong, Q. Tian, Z. Zhang et al., “HunyuanVideo: A Systematic Framework For Large Video Generative Models,” technical report, arXiv:2412.03603, 2024. report, official release.
  48. Team Wan, A. Wang, B. Ai, B. Wen et al., “Wan: Open and Advanced Large-Scale Video Generative Models,” technical report, arXiv:2503.20314, 2025. report, official Wan2.1 release.
  49. Z. Geng, Y. Lu, Z. Wu, E. Shechtman, J. Z. Kolter and K. He, “Improved Mean Flows: On the Challenges of Fastforward Generative Models,” CVPR, pp. 30467–30476, 2026. CVF
  50. Y. Lu, S. Lu, Q. Sun, H. Zhao, Z. Jiang, X. Wang, T. Li, Z. Geng and K. He, “One-step Latent-free Image Generation with Pixel Mean Flows,” ICML, PMLR 306, pp. 81265–81277, 2026. PMLR