JiT:在像素空間直接預測乾淨影像
選讀・ML 08・擴散模型與流匹配 的論文導讀(第 5/6 篇)。引用編號對應本頁底部的參考文獻。
導覽:本頁與本系列: 導覽:本頁與本系列
問題與論點
為何大區塊像素 Transformer 在直接預測乾淨影像時可行,改成直接預測雜訊或速度卻可能失敗?Li 與 He 主張,自然乾淨影像的結構維度遠低於周圍帶雜訊空間 [42]。Just image Transformers,簡稱 JiT,研究預測目標與網路容量的互動。先讀 ViT 區塊維度,以及正文中直接輸出與損失空間的區分。
方法與圖解

圖 3 包含像素區塊、線性嵌入、Transformer 區塊及影像區塊的線性預測;時間與類別作為條件。生成器沒有另一組潛在編碼器/解碼器。然而,在每個時間預測乾淨影像,不等於學到完整的一步生成映射。
為明示時間方向差異,以 表示 JiT 的時間:零端是雜訊,一端是資料。則
其中 為乾淨資料, 為先驗雜訊, 為插值輸入, 為網路直接輸出, 為導出的瞬時速度。式子在 、尚未加入數值保護時成立。換成 MeanFlow 慣例需代入 ,並反轉速度正負號。即使網路預測影像,求解器仍使用速度;速度損失等於時間加權影像誤差,不是未加權影像 MSE。
證據
表 2 交叉比較三種直接輸出與三種損失空間。JiT-B/16 在 256×256 ImageNet、200 個 epoch 與 CFG 下,速度損失那列的影像輸出 FID 為 8.62,雜訊輸出 372.38,速度輸出 96.53。較小的 64×64、區塊寬 4 設定,沒有同樣明顯的差距。固定損失後,可看出與容量相關的預測空間效果。圖 4 另顯示,影像預測搭配瓶頸區塊嵌入仍可有效。
設計分析
每個區塊的純量維度,不等於區塊數量。同時增加影像解析度及區塊大小,可固定 token 數,卻增加每個 token 所承載的資訊。人工嵌入實驗與影像消融,在有限網路容量下支持流形解釋,但沒有證明所有影像預測器都必然勝過所有雜訊預測器。只調損失權重或雜訊抽樣,也不能重現整個架構比較。
閱讀檢查點
分別指出直接輸出、損失目標,以及 ODE 求解器消耗的量。再解釋為何從高度帶雜訊輸入得到的一次乾淨預測,可能是多種可能性的平均,而非影像分布中的清晰樣本。
限制與下一步
JiT 研究類別條件生成,仍需數值取樣,本身不是一步或自由文字生成影像模型。它不使用 tokenizer 的配方,也不能與後續所有像素生成器是否使用預訓練損失混為一談。接著讀 Pixel MeanFlow,理解它如何加入雙時間去雜訊影像場及區間目標,以支援一次呼叫像素生成。
接著閱讀: Pixel MeanFlow:透過去雜訊影像場一步生成像素
參考文獻
- L. Yang, Z. Zhang, Y. Song, S. Hong, R. Xu, Y. Zhao, W. Zhang, B. Cui and M.-H. Yang, “Diffusion Models: A Comprehensive Survey of Methods and Applications,” ACM Computing Surveys, 2023. arXiv
- Y. Lipman, M. Havasi, P. Holderrieth, N. Shaul, M. Le, B. Karrer, R. T. Q. Chen, D. Lopez-Paz, H. Ben-Hamu and I. Gat, “Flow Matching Guide and Code,” arXiv:2412.06264, 2024. arXiv
- J. Ho, A. Jain and P. Abbeel, “Denoising Diffusion Probabilistic Models,” NeurIPS, 2020. arXiv
- Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon and B. Poole, “Score-Based Generative Modeling through Stochastic Differential Equations,” ICLR, 2021. arXiv
- J. Song, C. Meng and S. Ermon, “Denoising Diffusion Implicit Models,” ICLR, 2021. arXiv
- J. Ho and T. Salimans, “Classifier-Free Diffusion Guidance,” NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications; arXiv:2207.12598, 2022. arXiv
- R. Rombach, A. Blattmann, D. Lorenz, P. Esser and B. Ommer, “High-Resolution Image Synthesis with Latent Diffusion Models,” CVPR, 2022. arXiv
- W. Peebles and S. Xie, “Scalable Diffusion Models with Transformers,” ICCV, 2023. arXiv
- P. Esser, S. Kulal, A. Blattmann et al., “Scaling Rectified Flow Transformers for High-Resolution Image Synthesis,” ICML, PMLR 235, 2024. PMLR
- Y. Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel and M. Le, “Flow Matching for Generative Modeling,” ICLR, 2023. arXiv
- X. Liu, C. Gong and Q. Liu, “Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow,” ICLR, 2023. arXiv
- Y. Song, P. Dhariwal, M. Chen and I. Sutskever, “Consistency Models,” ICML, 2023. arXiv
- Z. Geng, M. Deng, X. Bai, J. Z. Kolter and K. He, “Mean Flows for One-step Generative Modeling,” NeurIPS, 2025. arXiv
- S. J. D. Prince, Understanding Deep Learning, MIT Press, 2023. book site
- J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan and S. Ganguli, “Deep Unsupervised Learning using Nonequilibrium Thermodynamics,” ICML, PMLR 37, 2015. PMLR
- P. Vincent, “A Connection Between Score Matching and Denoising Autoencoders,” Neural Computation, vol. 23, no. 7, pp. 1661–1674, 2011. doi
- Y. Song and S. Ermon, “Generative Modeling by Estimating Gradients of the Data Distribution,” NeurIPS, 2019. arXiv
- P. Dhariwal and A. Nichol, “Diffusion Models Beat GANs on Image Synthesis,” NeurIPS, 2021. NeurIPS
- A. Q. Nichol and P. Dhariwal, “Improved Denoising Diffusion Probabilistic Models,” ICML, PMLR 139, 2021. PMLR
- T. Karras, M. Aittala, T. Aila and S. Laine, “Elucidating the Design Space of Diffusion-Based Generative Models,” NeurIPS, 2022. arXiv
- C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li and J. Zhu, “DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps,” NeurIPS, 2022. arXiv
- T. Salimans and J. Ho, “Progressive Distillation for Fast Sampling of Diffusion Models,” ICLR, 2022. arXiv
- M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler and S. Hochreiter, “GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium,” NeurIPS (NIPS), 2017. arXiv
- T. Kynkäänniemi, T. Karras, S. Laine, J. Lehtinen and T. Aila, “Improved Precision and Recall Metric for Assessing Generative Models,” NeurIPS, 2019. arXiv
- G. Stein, J. C. Cresswell, R. Hosseinzadeh et al., “Exposing Flaws of Generative Model Evaluation Metrics and Their Unfair Treatment of Diffusion Models,” NeurIPS, 2023. arXiv
- S. Jayasumana, S. Ramalingam, A. Veit, D. Glasner, A. Chakrabarti and S. Kumar, “Rethinking FID: Towards a Better Evaluation Metric for Image Generation,” CVPR, 2024. CVF
- F.-A. Croitoru, V. Hondru, R. T. Ionescu and M. Shah, “Diffusion Models in Vision: A Survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023. doi
- H. Cao, C. Tan, Z. Gao, Y. Xu, G. Chen, P.-A. Heng and S. Z. Li, “A Survey on Generative Diffusion Models,” IEEE Transactions on Knowledge and Data Engineering, vol. 36, no. 7, pp. 2814–2830, 2024. doi
- C.-H. Lai, Y. Song, D. Kim, Y. Mitsufuji and S. Ermon, “The Principles of Diffusion Models,” monograph, arXiv:2510.21890, 2025. arXiv
- H. Shen, J. Zhang, B. Xiong et al., “Efficient Diffusion Models: A Survey,” Transactions on Machine Learning Research, 2025. arXiv
- A. Melnik, M. Ljubljanac, C. Lu, Q. Yan, W. Ren and H. Ritter, “Video Diffusion Models: A Survey,” Transactions on Machine Learning Research, 2024. arXiv
- C. Lu and Y. Song, “Simplifying, Stabilizing and Scaling Continuous-Time Consistency Models,” ICLR, 2025. arXiv
- A. Sauer, D. Lorenz, A. Blattmann and R. Rombach, “Adversarial Diffusion Distillation,” ECCV, 2024. arXiv
- M. S. Albergo and E. Vanden-Eijnden, “Building Normalizing Flows with Stochastic Interpolants,” ICLR, 2023. arXiv
- J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li and L. Fei-Fei, “ImageNet: A Large-Scale Hierarchical Image Database,” CVPR, 2009, pp. 248–255. doi
- C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens and Z. Wojna, “Rethinking the Inception Architecture for Computer Vision,” CVPR, 2016, pp. 2818–2826. arXiv
- O. Ronneberger, P. Fischer and T. Brox, “U-Net: Convolutional Networks for Biomedical Image Segmentation,” MICCAI, LNCS 9351, pp. 234–241, 2015. arXiv
- A. Radford, J. W. Kim, C. Hallacy et al., “Learning Transferable Visual Models From Natural Language Supervision,” ICML, 2021. arXiv
- M. Oquab, T. Darcet, T. Moutakanni et al., “DINOv2: Learning Robust Visual Features without Supervision,” Transactions on Machine Learning Research, 2024. arXiv
- B. Efron, “Tweedie’s Formula and Selection Bias,” Journal of the American Statistical Association, vol. 106, no. 496, pp. 1602–1614, 2011. doi
- A. Dosovitskiy, L. Beyer, A. Kolesnikov et al., “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale,” ICLR, 2021. arXiv
- T. Li and K. He, “Back to Basics: Let Denoising Generative Models Denoise,” CVPR, 2026 (preprint 2025). paper, publication record, official JiT code.
- B. Zheng, N. Ma, S. Tong and S. Xie, “Diffusion Transformers with Representation Autoencoders,” ICLR, 2026 (preprint 2025). proceedings, manuscript.
- S. Tong, B. Zheng, Z. Wang, B. Tang, N. Ma, E. Brown, J. Yang, R. Fergus, Y. LeCun and S. Xie, “Scaling Text-to-Image Diffusion Transformers with Representation Autoencoders,” CVPR Findings Track, 2026. manuscript, author publication record.
- D. Podell, Z. English, K. Lacey, A. Blattmann, T. Dockhorn, J. Müller, J. Penna and R. Rombach, “SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis,” ICLR, 2024. proceedings, official model card.
- Black Forest Labs, “FLUX.1 [dev],” official pretrained-model card, 2024. model card.
- W. Kong, Q. Tian, Z. Zhang et al., “HunyuanVideo: A Systematic Framework For Large Video Generative Models,” technical report, arXiv:2412.03603, 2024. report, official release.
- Team Wan, A. Wang, B. Ai, B. Wen et al., “Wan: Open and Advanced Large-Scale Video Generative Models,” technical report, arXiv:2503.20314, 2025. report, official Wan2.1 release.
- Z. Geng, Y. Lu, Z. Wu, E. Shechtman, J. Z. Kolter and K. He, “Improved Mean Flows: On the Challenges of Fastforward Generative Models,” CVPR, pp. 30467–30476, 2026. CVF
- Y. Lu, S. Lu, Q. Sun, H. Zhao, Z. Jiang, X. Wang, T. Li, Z. Geng and K. He, “One-step Latent-free Image Generation with Pixel Mean Flows,” ICML, PMLR 306, pp. 81265–81277, 2026. PMLR