完整搜尋頁面 →

論文導讀

選讀:DLCV 12・影像生成:從 GAN 到文字生成影像系統 所依據論文的導讀,每篇一頁:說明它要解決的問題、核心想法、結果,以及閱讀時該注意什麼。

導覽:本頁與本系列: 導覽:本頁與本系列

本頁內容

補充資料 · 4

  1. 統一多模態模型逐一介紹
  2. ControlNet 的零卷積:為什麼訓練從原始模型開始
  3. 個人化目標函數並排比較
  4. 硬綁定、軟綁定與浮水印

論文導讀 · 3

  1. Latent Diffusion:先壓縮感知資訊,再學習影像分布
  2. Esser 等人:Scaling Rectified Flow Transformers for High-Resolution Image Synthesis(ICML 2024)
  3. Ghosh 等人:GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment(NeurIPS 2023)

電腦視覺深度學習

  1. 02 · 深度影像分類
  2. 03 · 物件偵測:從 R-CNN 到 DETR
  3. 09 · 深度影像復原:低階視覺
查看完整課程樹 →

這些導讀是選讀內容:每篇帶你讀一篇本篇依據的論文——它要解決的問題、核心想法、它呈現了什麼,以及閱讀時該檢查什麼。

  1. Latent Diffusion:先壓縮感知資訊,再學習影像分布

    像素空間擴散反覆處理大型網格,即使其中許多細節可以更精簡地表示。

  2. Esser 等人:Scaling Rectified Flow Transformers for High-Resolution Image Synthesis(ICML 2024)

    Stable Diffusion 3 的研究論文 [20] 問的是:什麼訓練目標、時間取樣、表示與骨幹組合,能有效擴展文字生成影像?

  3. Ghosh 等人:GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment(NeurIPS 2023)

    FID 衡量整體的影像統計,CLIPScore 衡量整體的影像–文字相似度;兩者都無法告訴你模型是否畫了兩隻狗、一輛紅車,或在筆電左邊的杯子。

參考文獻

  1. T. Karras, S. Laine and T. Aila, “A Style-Based Generator Architecture for Generative Adversarial Networks,” CVPR, 2019. arXiv
  2. A. Brock, J. Donahue and K. Simonyan, “Large Scale GAN Training for High Fidelity Natural Image Synthesis,” ICLR, 2019. arXiv
  3. W. Peebles and S. Xie, “Scalable Diffusion Models with Transformers,” ICCV, 2023. CVF
  4. C. Saharia, W. Chan, S. Saxena et al., “Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding,” NeurIPS, 2022. arXiv · project page
  5. R. Rombach, A. Blattmann, D. Lorenz, P. Esser and B. Ommer, “High-Resolution Image Synthesis with Latent Diffusion Models,” CVPR, 2022. CVF
  6. C. Meng, Y. He, Y. Song et al., “SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations,” ICLR, 2022. arXiv
  7. L. Zhang, A. Rao and M. Agrawala, “Adding Conditional Control to Text-to-Image Diffusion Models,” ICCV, 2023. CVF
  8. A. Hertz, R. Mokady, J. Tenenbaum et al., “Prompt-to-Prompt Image Editing with Cross-Attention Control,” ICLR, 2023. arXiv
  9. T. Brooks, A. Holynski and A. A. Efros, “InstructPix2Pix: Learning to Follow Image Editing Instructions,” CVPR, 2023. CVF
  10. A. Lugmayr, M. Danelljan, A. Romero et al., “RePaint: Inpainting using Denoising Diffusion Probabilistic Models,” CVPR, 2022. CVF
  11. J. Ho and T. Salimans, “Classifier-Free Diffusion Guidance,” NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications; arXiv:2207.12598, 2022. arXiv
  12. I. J. Goodfellow, J. Pouget-Abadie, M. Mirza et al., “Generative Adversarial Nets,” NeurIPS, 2014. arXiv
  13. D. P. Kingma and M. Welling, “Auto-Encoding Variational Bayes,” ICLR, 2014. arXiv
  14. A. van den Oord, O. Vinyals and K. Kavukcuoglu, “Neural Discrete Representation Learning,” NeurIPS, 2017. arXiv
  15. P. Esser, R. Rombach and B. Ommer, “Taming Transformers for High-Resolution Image Synthesis,” CVPR, 2021. CVF
  16. J. Ho, A. Jain and P. Abbeel, “Denoising Diffusion Probabilistic Models,” NeurIPS, 2020. arXiv
  17. P. Dhariwal and A. Nichol, “Diffusion Models Beat GANs on Image Synthesis,” NeurIPS, 2021. arXiv
  18. Y. Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel and M. Le, “Flow Matching for Generative Modeling,” ICLR, 2023. arXiv
  19. X. Liu, C. Gong and Q. Liu, “Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow,” ICLR, 2023. arXiv
  20. P. Esser, S. Kulal, A. Blattmann et al., “Scaling Rectified Flow Transformers for High-Resolution Image Synthesis,” ICML, PMLR 235, 2024. arXiv
  21. T. Li, Y. Tian, H. Li, M. Deng and K. He, “Autoregressive Image Generation without Vector Quantization,” NeurIPS, 2024. arXiv
  22. A. Sauer, D. Lorenz, A. Blattmann and R. Rombach, “Adversarial Diffusion Distillation,” ECCV, LNCS, pp. 87–103, 2024. doi · arXiv
  23. A. Radford, J. W. Kim, C. Hallacy et al., “Learning Transferable Visual Models From Natural Language Supervision,” ICML, PMLR 139, 2021. arXiv
  24. C. Raffel, N. Shazeer, A. Roberts et al., “Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer,” Journal of Machine Learning Research, vol. 21, 2020. arXiv
  25. D. Podell, Z. English, K. Lacey et al., “SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis,” ICLR, 2024. arXiv
  26. R. Liu, D. Garrette, C. Saharia et al., “Character-Aware Models Improve Visual Text Rendering,” ACL, 2023. ACL Anthology
  27. A. Ramesh, P. Dhariwal, A. Nichol, C. Chu and M. Chen, “Hierarchical Text-Conditional Image Generation with CLIP Latents,” arXiv:2204.06125, 2022. arXiv
  28. H. Chefer, Y. Alaluf, Y. Vinker, L. Wolf and D. Cohen-Or, “Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models,” ACM Transactions on Graphics, vol. 42, no. 4 (SIGGRAPH), 2023. doi · arXiv
  29. O. Ronneberger, P. Fischer and T. Brox, “U-Net: Convolutional Networks for Biomedical Image Segmentation,” MICCAI, 2015. arXiv
  30. J. Yu, Y. Xu, J. Y. Koh et al., “Scaling Autoregressive Models for Content-Rich Text-to-Image Generation,” Transactions on Machine Learning Research, 2022. arXiv
  31. H. Chang, H. Zhang, J. Barber et al., “Muse: Text-To-Image Generation via Masked Generative Transformers,” ICML, PMLR 202, 2023. arXiv
  32. R. Gal, Y. Alaluf, Y. Atzmon et al., “An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion,” ICLR, 2023. arXiv
  33. N. Ruiz, Y. Li, V. Jampani, Y. Pritch, M. Rubinstein and K. Aberman, “DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation,” CVPR, 2023. CVF
  34. E. J. Hu, Y. Shen, P. Wallis et al., “LoRA: Low-Rank Adaptation of Large Language Models,” ICLR, 2022. arXiv
  35. Y. Huang, J. Huang, Y. Liu et al., “Diffusion Model-Based Image Editing: A Survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025. doi · arXiv
  36. J. Song, C. Meng and S. Ermon, “Denoising Diffusion Implicit Models,” ICLR, 2021. arXiv
  37. T. Salimans and J. Ho, “Progressive Distillation for Fast Sampling of Diffusion Models,” ICLR, 2022. arXiv
  38. Y. Song, P. Dhariwal, M. Chen and I. Sutskever, “Consistency Models,” ICML, PMLR 202, 2023. arXiv
  39. P. Sun, Y. Jiang, S. Chen et al., “Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation,” arXiv:2406.06525, 2024. arXiv
  40. H. Chang, H. Zhang, L. Jiang, C. Liu and W. T. Freeman, “MaskGIT: Masked Generative Image Transformer,” CVPR, 2022. CVF
  41. K. Tian, Y. Jiang, Z. Yuan, B. Peng and L. Wang, “Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction,” NeurIPS, 2024. arXiv
  42. L. Yu, J. Lezama, N. B. Gundavarapu et al., “Language Model Beats Diffusion — Tokenizer is Key to Visual Generation,” ICLR, 2024. arXiv
  43. Chameleon Team, “Chameleon: Mixed-Modal Early-Fusion Foundation Models,” arXiv:2405.09818, 2024. arXiv
  44. X. Wang, Y. Cui, J. Wang et al., “Multimodal learning with next-token prediction for large multimodal models,” Nature, vol. 650, pp. 327–333, 2026 (preprint “Emu3: Next-Token Prediction is All You Need,” 2024). doi · arXiv
  45. C. Zhou, L. Yu, A. Babu et al., “Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model,” ICLR, 2025. arXiv
  46. J. Xie, W. Mao, Z. Bai et al., “Show-o: One Single Transformer to Unify Multimodal Understanding and Generation,” ICLR, 2025. arXiv
  47. C. Wu, X. Chen, Z. Wu et al., “Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation,” CVPR, 2025. CVF
  48. T.-Y. Lin, M. Maire, S. Belongie et al., “Microsoft COCO: Common Objects in Context,” ECCV, 2014. arXiv
  49. M. J. Chong and D. Forsyth, “Effectively Unbiased FID and Inception Score and Where to Find Them,” CVPR, 2020. arXiv
  50. G. Parmar, R. Zhang and J.-Y. Zhu, “On Aliased Resizing and Surprising Subtleties in GAN Evaluation,” CVPR, 2022. CVF
  51. T. Kynkäänniemi, T. Karras, M. Aittala, T. Aila and J. Lehtinen, “The Role of ImageNet Classes in Fréchet Inception Distance,” ICLR, 2023. arXiv
  52. G. Stein, J. C. Cresswell, R. Hosseinzadeh et al., “Exposing Flaws of Generative Model Evaluation Metrics and Their Unfair Treatment of Diffusion Models,” NeurIPS, 2023. arXiv
  53. S. Jayasumana, S. Ramalingam, A. Veit, D. Glasner, A. Chakrabarti and S. Kumar, “Rethinking FID: Towards a Better Evaluation Metric for Image Generation,” CVPR, 2024. CVF
  54. J. Hessel, A. Holtzman, M. Forbes, R. Le Bras and Y. Choi, “CLIPScore: A Reference-free Evaluation Metric for Image Captioning,” EMNLP, 2021. ACL Anthology
  55. D. Ghosh, H. Hajishirzi and L. Schmidt, “GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment,” NeurIPS Datasets and Benchmarks Track, 2023. arXiv
  56. K. Huang, K. Sun, E. Xie, Z. Li and X. Liu, “T2I-CompBench: A Comprehensive Benchmark for Open-world Compositional Text-to-image Generation,” NeurIPS Datasets and Benchmarks Track, 2023. proceedings
  57. J. Xu, X. Liu, Y. Wu et al., “ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation,” NeurIPS, 2023. arXiv
  58. Y. Kirstain, A. Polyak, U. Singer et al., “Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation,” NeurIPS, 2023. arXiv
  59. T. Lee, M. Yasunaga, C. Meng et al., “Holistic Evaluation of Text-To-Image Models,” NeurIPS Datasets and Benchmarks Track, 2023. arXiv
  60. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li and L. Fei-Fei, “ImageNet: A Large-Scale Hierarchical Image Database,” CVPR, 2009. doi
  61. C. Schuhmann, R. Beaumont, R. Vencu et al., “LAION-5B: An Open Large-Scale Dataset for Training Next Generation Image-Text Models,” NeurIPS Datasets and Benchmarks Track, 2022. arXiv
  62. A. Birhane, V. U. Prabhu and E. Kahembwe, “Multimodal datasets: misogyny, pornography, and malignant stereotypes,” arXiv:2110.01963, 2021. arXiv
  63. N. Carlini, J. Hayes, M. Nasr et al., “Extracting Training Data from Diffusion Models,” USENIX Security Symposium, 2023. USENIX · arXiv
  64. G. Somepalli, V. Singla, M. Goldblum, J. Geiping and T. Goldstein, “Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models,” CVPR, 2023. CVF
  65. F. Bianchi, P. Kalluri, E. Durmus et al., “Easily Accessible Text-to-Image Generation Amplifies Demographic Stereotypes at Large Scale,” ACM FAccT, 2023. doi
  66. A. S. Luccioni, C. Akiki, M. Mitchell and Y. Jernite, “Stable Bias: Analyzing Societal Representations in Diffusion Models,” NeurIPS Datasets and Benchmarks Track, 2023. arXiv
  67. Coalition for Content Provenance and Authenticity (C2PA), “C2PA Technical Specification,” version 2.1, 2024. specification
  68. Y. Wen, J. Kirchenbauer, J. Geiping and T. Goldstein, “Tree-Ring Watermarks: Fingerprints for Diffusion Images that are Invisible and Robust,” NeurIPS, 2023. arXiv
  69. P. Fernandez, G. Couairon, H. Jégou, M. Douze and T. Furon, “The Stable Signature: Rooting Watermarks in Latent Diffusion Models,” ICCV, 2023. CVF
  70. X. Zhao, K. Zhang, Z. Su et al., “Invisible Image Watermarks Are Provably Removable Using Generative AI,” NeurIPS, 2024. arXiv
  71. M. Saberi, V. S. Sadasivan, K. Rezaei et al., “Robustness of AI-Image Detectors: Fundamental Limits and Practical Attacks,” ICLR, 2024. arXiv
  72. W. Lu, Y. Xu, J. Zhang, C. Wang and D. Tao, “HandRefiner: Refining Malformed Hands in Generated Images by Diffusion-based Conditional Inpainting,” ACM Multimedia, 2024, pp. 7085–7093. doi
  73. C. Zhang, C. Zhang, M. Zhang, I. S. Kweon and J. Kim, “Text-to-image Diffusion Models in Generative AI: A Survey,” arXiv:2303.07909, 2023. arXiv
  74. F.-A. Croitoru, V. Hondru, R. T. Ionescu and M. Shah, “Diffusion Models in Vision: A Survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023. doi
  75. J. Xiong, G. Liu, L. Huang et al., “Autoregressive Models in Vision: A Survey,” Transactions on Machine Learning Research, 2025. arXiv