完整搜尋頁面 →

Chung 等人:Diffusion Posterior Sampling for General Noisy Inverse Problems(ICLR 2023)

選讀・DLCV 09B・生成式影像復原:取樣出一個合理的答案 的論文導讀(第 2/5 篇)。引用編號對應本頁底部的參考文獻。

導覽:本頁與本系列: 導覽:本頁與本系列

本頁內容

補充資料 · 4

  1. 為什麼後驗取樣恰好讓 MSE 變成兩倍
  2. 以 GAN 為先驗的復原方法詳解
  3. 零樣本擴散取樣器有何不同
  4. 潛在擴散與一步復原方法逐一介紹

論文導讀 · 5

  1. Blau 與 Michaeli:The Perception-Distortion Tradeoff(CVPR 2018)
  2. Chung 等人:Diffusion Posterior Sampling for General Noisy Inverse Problems(ICLR 2023)
  3. DDNM:保留量測成分,生成零空間內容
  4. BlindDPS:共同推論影像與退化
  5. Li 等人:Diffusion Models for Image Restoration and Enhancement: A Comprehensive Survey(IJCV 2025)

電腦視覺深度學習

  1. 02 · 深度影像分類
  2. 03 · 物件偵測:從 R-CNN 到 DETR
  3. 09 · 深度影像復原:低階視覺
查看完整課程樹 →

挑戰

無條件擴散能產生可信影像,卻未必符合指定量測。DPS [17] 使用已知量測算子與雜訊模型,在取樣時引導先驗。目標是條件分布,而非只找出量測殘差最小的照片。

方法

此機率圖區別可計算的乾淨影像概似與通常難算的帶雜訊狀態概似;下文推導說明如何近似。
原論文圖 2 — 此機率圖區別可計算的乾淨影像概似與通常難算的帶雜訊狀態概似;下文推導說明如何近似。

圖片來源: [17], PDF 第 4 頁. 原始 PDF.

分開影像先驗與量測概似。 對乾淨影像 x0x_0、量測 yy、已知且可能非線性的算子 A\mathcal A,假設 y=A(x0)+ny=\mathcal A(x_0)+n。若 n∼N(0,σy2I)n\sim\mathcal N(0,\sigma_y^2I),σy\sigma_y 為量測雜訊標準差,II 是單位矩陣。無條件分數網路估計帶雜訊影像的分數 sθ(xt,t)s_\theta(x_t,t)。

乾淨影像估計為

x^0(xt)=xt+(1−αˉt)sθ(xt,t)αˉt.\widehat x_0(x_t)= \frac{x_t+(1-\bar\alpha_t)s_\theta(x_t,t)} {\sqrt{\bar\alpha_t}}.

xtx_t 是帶雜訊的取樣狀態,αˉt\bar\alpha_t 是保留訊號功率比例。此式使用變異數保持的分數參數化;預測雜訊的實作必須一致地轉換網路輸出。

困難的 p(y∣xt)p(y\mid x_t) 要對可能的乾淨影像積分。DPS 以 x^0(xt)\widehat x_0(x_t) 代入乾淨影像概似來近似。定義

ℓt=∥y−A(x^0(xt))∥222σy2,spost(xt,t)≈sθ(xt,t)−∇xtℓt.\ell_t=\frac{\|y-\mathcal A(\widehat x_0(x_t))\|_2^2}{2\sigma_y^2}, \qquad s_{\rm post}(x_t,t)\approx s_\theta(x_t,t)-\nabla_{x_t}\ell_t.

ℓt\ell_t 是略去常數的高斯負對數概似,sposts_{\rm post} 是近似條件分數。梯度必須通過量測算子與去雜訊器的乾淨影像估計兩者;若切斷後者梯度,就改變了演算法。

交替做先驗移動與概似修正。 實際離散演算法先提出反向擴散狀態,再以當前狀態的量測殘差梯度修正。依殘差正規化的步長是實務穩定化規則,不應把程式係數誤認為直接保留未修改的解析 1/σy21/\sigma_y^2。

Poisson 觀測使用依量測而定的加權殘差近似,不是換個資料集名稱就沿用高斯概似;低計數情況須特別小心。原文的相位恢復等可微非線性算子,也能不計算 SVD 而納入。

DPS 計算流程:乾淨估計產生量測殘差,以其梯度修正反向擴散步驟
教學示意圖 — 上方原論文機率圖未畫出計算路徑;量測梯度穿過去雜訊器與已知算子,再修正下一個狀態。這是解說用抽象圖,不是新的取樣演算法。

讀圖與選用判斷。 從帶雜訊的狀態追到乾淨估計,對估計套用已知前向算子,與量測比較,再把差異的梯度傳回帶雜訊的狀態。殘差在量測空間計算,不是推論時能取得的乾淨影像誤差。可微分算子讓修正成為可能,但不會使近似後驗自動變成精確後驗。尤其在含雜訊資料上,要同時檢查量測吻合與重建合理性;強迫殘差歸零可能是在擬合雜訊。因此,步長強度與似然模型都必須納入可重現的比較。

結果

表 1 評估 256×256256\times256 FFHQ 帶雜訊修復,高斯標準差為 0.05。4×4\times SR 的 DPS 為 FID 39.35/LPIPS 0.214,同實驗 DDRM 為 62.15/0.294。不能直接與無雜訊 DDNM 表格比較,彷彿量測模型沒有改變。

原文也評估 ImageNet、修補、高斯/運動去模糊與非線性問題。附錄回報 RTX 2080 Ti、1,000 次網路評估下,每張 FFHQ 約 95 秒、ImageNet 約 600 秒。這是歷史實驗的完整設定,不是對當前裝置的保證。

消融實驗

附錄 C 改變概似步長強度:太小會不符合量測,太大則造成飽和、雜訊放大。測試中正規化強度約 0.1–1.0 較實用,並非所有算子共用的普遍數值。

表 5 的 100 張 FFHQ 高斯去模糊研究,提出的殘差正規化規則為 LPIPS 0.247 ± 0.045,測試的逆雜訊變異數配置為 0.727 ± 0.038;線性、指數遞減也有差異。圖 11 比較函數評估數與實際時間,說明品質宣稱必須附取樣預算。

結論

DPS 透過乾淨影像估計反傳概似梯度,把可重用影像先驗變成彈性的帶雜訊逆問題解法。代價是反覆網路評估、反向傳播,以及對真實條件分數的近似。

限制與問題

代入式概似通常不等於真正邊際概似,因此不保證精確後驗取樣。結果取決於分數品質、算子/雜訊模型與引導強度。小殘差不保證不可見細節正確;相位恢復取多樣本最佳結果,也不同於只執行一次。未知算子參數須另加估計與先驗機制,例如 BlindDPS。

參考文獻

  1. Y. Blau and T. Michaeli, “The Perception-Distortion Tradeoff,” in Proc. IEEE/CVF CVPR, 2018, pp. 6228–6237 (extended version on arXiv). doi · arXiv
  2. C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang and W. Shi, “Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network,” in Proc. IEEE CVPR, 2017, pp. 105–114. doi · arXiv
  3. J. Johnson, A. Alahi and L. Fei-Fei, “Perceptual Losses for Real-Time Style Transfer and Super-Resolution,” in Proc. ECCV, 2016, pp. 694–711. doi · arXiv
  4. X. Wang, K. Yu, S. Wu, J. Gu, Y. Liu, C. Dong, Y. Qiao and C. C. Loy, “ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks,” in Proc. ECCV Workshops, 2018 (LNCS, 2019), pp. 63–79. doi · arXiv
  5. X. Wang, L. Xie, C. Dong and Y. Shan, “Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data,” in Proc. IEEE/CVF ICCV Workshops, 2021, pp. 1905–1914. doi · arXiv
  6. T. Karras, S. Laine and T. Aila, “A Style-Based Generator Architecture for Generative Adversarial Networks,” in Proc. IEEE/CVF CVPR, 2019, pp. 4396–4405. doi · arXiv
  7. S. Menon, A. Damian, S. Hu, N. Ravi and C. Rudin, “PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Models,” in Proc. IEEE/CVF CVPR, 2020, pp. 2434–2442. doi · arXiv
  8. K. C. K. Chan, X. Wang, X. Xu, J. Gu and C. C. Loy, “GLEAN: Generative Latent Bank for Large-Factor Image Super-Resolution,” in Proc. IEEE/CVF CVPR, 2021, pp. 14240–14249. doi · arXiv
  9. X. Wang, Y. Li, H. Zhang and Y. Shan, “Towards Real-World Blind Face Restoration with Generative Facial Prior,” in Proc. IEEE/CVF CVPR, 2021, pp. 9164–9174. doi · arXiv
  10. S. Zhou, K. C. K. Chan, C. Li and C. C. Loy, “Towards Robust Blind Face Restoration with Codebook Lookup Transformer,” in Proc. NeurIPS, vol. 35, 2022, pp. 30599–30611. doi · arXiv
  11. J. Ho, A. Jain and P. Abbeel, “Denoising Diffusion Probabilistic Models,” in Proc. NeurIPS, 2020. proceedings · arXiv
  12. Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon and B. Poole, “Score-Based Generative Modeling through Stochastic Differential Equations,” in Proc. ICLR, 2021. arXiv
  13. B. Efron, “Tweedie’s Formula and Selection Bias,” Journal of the American Statistical Association, vol. 106, no. 496, pp. 1602–1614, 2011. doi
  14. C. Saharia, J. Ho, W. Chan, T. Salimans, D. J. Fleet and M. Norouzi, “Image Super-Resolution via Iterative Refinement,” IEEE TPAMI, 2022 (early access). doi · arXiv
  15. C. Saharia, W. Chan, H. Chang, C. Lee, J. Ho, T. Salimans, D. Fleet and M. Norouzi, “Palette: Image-to-Image Diffusion Models,” in Proc. ACM SIGGRAPH Conference, 2022, pp. 1–10. doi · arXiv
  16. Z. Yue, J. Wang and C. C. Loy, “ResShift: Efficient Diffusion Model for Image Super-resolution by Residual Shifting,” in Proc. NeurIPS, vol. 36, 2023, pp. 13294–13307. doi · arXiv
  17. H. Chung, J. Kim, M. T. McCann, M. L. Klasky and J. C. Ye, “Diffusion Posterior Sampling for General Noisy Inverse Problems,” in Proc. ICLR, 2023. arXiv
  18. B. Kawar, M. Elad, S. Ermon and J. Song, “Denoising Diffusion Restoration Models,” in Proc. NeurIPS, vol. 35, 2022, pp. 23593–23606. doi · arXiv
  19. Y. Wang, J. Yu and J. Zhang, “Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Model,” in Proc. ICLR, 2023. arXiv
  20. J. Song, A. Vahdat, M. Mardani and J. Kautz, “Pseudoinverse-Guided Diffusion Models for Inverse Problems,” in Proc. ICLR, 2023. ICLR page
  21. Y. Zhu, K. Zhang, J. Liang, J. Cao, B. Wen, R. Timofte and L. Van Gool, “Denoising Diffusion Models for Plug-and-Play Image Restoration,” in Proc. IEEE/CVF CVPR Workshops, 2023, pp. 1219–1229. doi · arXiv
  22. R. Rombach, A. Blattmann, D. Lorenz, P. Esser and B. Ommer, “High-Resolution Image Synthesis with Latent Diffusion Models,” in Proc. IEEE/CVF CVPR, 2022, pp. 10674–10685. doi · arXiv
  23. L. Zhang, A. Rao and M. Agrawala, “Adding Conditional Control to Text-to-Image Diffusion Models,” in Proc. IEEE/CVF ICCV, 2023, pp. 3813–3824. doi · arXiv
  24. J. Wang, Z. Yue, S. Zhou, K. C. K. Chan and C. C. Loy, “Exploiting Diffusion Prior for Real-World Image Super-Resolution,” International Journal of Computer Vision, vol. 132, no. 12, pp. 5929–5949, 2024. doi · arXiv
  25. X. Lin, J. He, Z. Chen, Z. Lyu, B. Dai, F. Yu, Y. Qiao, W. Ouyang and C. Dong, “DiffBIR: Toward Blind Image Restoration with Generative Diffusion Prior,” in Proc. ECCV, 2024 (LNCS), pp. 430–448. doi · arXiv
  26. F. Yu, J. Gu, Z. Li, J. Hu, X. Kong, X. Wang, J. He, Y. Qiao and C. Dong, “Scaling Up to Excellence: Practicing Model Scaling for Photo-Realistic Image Restoration In the Wild,” in Proc. IEEE/CVF CVPR, 2024, pp. 25669–25680. doi · arXiv
  27. Y. Wang, W. Yang, X. Chen, Y. Wang, L. Guo, L.-P. Chau, Z. Liu, Y. Qiao, A. C. Kot and B. Wen, “SinSR: Diffusion-Based Image Super-Resolution in a Single Step,” in Proc. IEEE/CVF CVPR, 2024, pp. 25796–25805. doi · arXiv
  28. R. Wu, L. Sun, Z. Ma and L. Zhang, “One-Step Effective Diffusion Network for Real-World Image Super-Resolution,” in Proc. NeurIPS, vol. 37, 2024, pp. 92529–92553. doi · arXiv
  29. Z. Yue, K. Liao and C. C. Loy, “Arbitrary-steps Image Super-resolution via Diffusion Inversion,” in Proc. IEEE/CVF CVPR, 2025, pp. 23153–23163. doi · arXiv
  30. L. Dong, Q. Fan, Y. Guo, Z. Wang, Q. Zhang, J. Chen, Y. Luo and C. Zou, “TSD-SR: One-Step Diffusion with Target Score Distillation for Real-World Image Super-Resolution,” in Proc. IEEE/CVF CVPR, 2025, pp. 23174–23184. doi · arXiv
  31. Y. Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel and M. Le, “Flow Matching for Generative Modeling,” in Proc. ICLR, 2023. arXiv
  32. X. Liu, C. Gong and Q. Liu, “Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow,” in Proc. ICLR, 2023. arXiv
  33. G. Ohayon, T. Michaeli and M. Elad, “Posterior-Mean Rectified Flow: Towards Minimum MSE Photo-Realistic Image Restoration,” in Proc. ICLR, 2025. arXiv
  34. S. Martin, A. Gagneux, P. Hagemann and G. Steidl, “PnP-Flow: Plug-and-Play Image Restoration with Flow Matching,” in Proc. ICLR, 2025. arXiv
  35. S. Bhadra, V. A. Kelkar, F. J. Brooks and M. A. Anastasio, “On Hallucinations in Tomographic Image Reconstruction,” IEEE Transactions on Medical Imaging, vol. 40, no. 11, pp. 3249–3260, 2021. doi · arXiv
  36. W. Ren, R. Goyal, Z. Hu, T. T. Aumentado-Armstrong, I. Mohomed and A. Levinshtein, “Hallucination Score: Towards Mitigating Hallucinations in Generative Image Super-Resolution,” arXiv:2507.14367, 2025. arXiv
  37. A. N. Angelopoulos, A. P. Kohli, S. Bates, M. I. Jordan, J. Malik, T. Alshaabi, S. Upadhyayula and Y. Romano, “Image-to-Image Regression with Distribution-Free Uncertainty Quantification and Applications in Imaging,” in Proc. ICML, PMLR 162, 2022. proceedings · arXiv
  38. R. Zhang, P. Isola, A. A. Efros, E. Shechtman and O. Wang, “The Unreasonable Effectiveness of Deep Features as a Perceptual Metric,” in Proc. IEEE/CVF CVPR, 2018, pp. 586–595. doi · arXiv
  39. Z. Wang, A. C. Bovik, H. R. Sheikh and E. P. Simoncelli, “Image Quality Assessment: From Error Visibility to Structural Similarity,” IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600–612, 2004. doi
  40. M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler and S. Hochreiter, “GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium,” in Proc. NeurIPS (NIPS), 2017. arXiv
  41. A. Mittal, R. Soundararajan and A. C. Bovik, “Making a ‘Completely Blind’ Image Quality Analyzer,” IEEE Signal Processing Letters, vol. 20, no. 3, pp. 209–212, 2013. doi
  42. J. Ke, Q. Wang, Y. Wang, P. Milanfar and F. Yang, “MUSIQ: Multi-scale Image Quality Transformer,” in Proc. IEEE/CVF ICCV, 2021, pp. 5128–5137. doi · arXiv
  43. J. Wang, K. C. K. Chan and C. C. Loy, “Exploring CLIP for Assessing the Look and Feel of Images,” in Proc. AAAI, vol. 37, no. 2, 2023, pp. 2555–2563. doi · arXiv
  44. X. Li, Y. Ren, X. Jin, C. Lan, X. Wang, W. Zeng, X. Wang and Z. Chen, “Diffusion Models for Image Restoration and Enhancement: A Comprehensive Survey,” International Journal of Computer Vision, vol. 133, no. 11, pp. 8078–8108, 2025. doi · arXiv
  45. G. Daras, H. Chung, C.-H. Lai, Y. Mitsufuji, J. C. Ye, P. Milanfar, A. G. Dimakis and M. Delbracio, “A Survey on Diffusion Models for Inverse Problems,” arXiv:2410.00083, 2024. arXiv
  46. B. B. Moser, A. S. Shanbhag, F. Raue, S. Frolov, S. Palacio and A. Dengel, “Diffusion Models, Image Super-Resolution, and Everything: A Survey,” IEEE Transactions on Neural Networks and Learning Systems, vol. 36, no. 7, pp. 11793–11813, 2025. doi · arXiv
  47. W. Xia, Y. Zhang, Y. Yang, J.-H. Xue, B. Zhou and M.-H. Yang, “GAN Inversion: A Survey,” IEEE TPAMI, vol. 45, no. 3, pp. 3121–3138, 2023. doi · arXiv
  48. Y. Song and S. Ermon, “Generative Modeling by Estimating Gradients of the Data Distribution,” in Proc. NeurIPS, 2019. arXiv
  49. H. Chung, J. Kim, S. Kim and J. C. Ye, “Parallel Diffusion Models of Operator and Image for Blind Inverse Problems,” in Proc. IEEE/CVF CVPR, 2023, pp. 6059–6069. paper