完整搜尋頁面 →

BlindDPS:共同推論影像與退化

選讀・DLCV 09B・生成式影像復原:取樣出一個合理的答案 的論文導讀(第 4/5 篇)。引用編號對應本頁底部的參考文獻。

導覽:本頁與本系列: 導覽:本頁與本系列

本頁內容

補充資料 · 4

  1. 為什麼後驗取樣恰好讓 MSE 變成兩倍
  2. 以 GAN 為先驗的復原方法詳解
  3. 零樣本擴散取樣器有何不同
  4. 潛在擴散與一步復原方法逐一介紹

論文導讀 · 5

  1. Blau 與 Michaeli:The Perception-Distortion Tradeoff(CVPR 2018)
  2. Chung 等人:Diffusion Posterior Sampling for General Noisy Inverse Problems(ICLR 2023)
  3. DDNM:保留量測成分,生成零空間內容
  4. BlindDPS:共同推論影像與退化
  5. Li 等人:Diffusion Models for Image Restoration and Enhancement: A Comprehensive Survey(IJCV 2025)

電腦視覺深度學習

  1. 02 · 深度影像分類
  2. 03 · 物件偵測:從 R-CNN 到 DETR
  3. 09 · 深度影像復原:低階視覺
查看完整課程樹 →

挑戰

非盲 DPS 假設已知前向算子。BlindDPS,即 CVPR 2023 的「Parallel Diffusion Models of Operator and Image for Blind Inverse Problems」[49],還要估計算子參數。運動去模糊中的清晰影像、模糊核都未知;沒有適當先驗,只降低共同殘差不足以識別兩者。

方法

兩個去雜訊器分別估計影像與算子,共同殘差耦合更新,而不是輸入已知模糊核。
原論文圖 2 — 兩個去雜訊器分別估計影像與算子,共同殘差耦合更新,而不是輸入已知模糊核。

圖片來源: [49], PDF 第 3 頁. 原始 PDF.

明確列出未知量。 對影像 xx、模糊核 kk、卷積 ∗*、量測雜訊 nn,基本模型為

y=k∗x+n,p(x,k∣y)∝p(y∣x,k) p(x) p(k).y=k*x+n,\qquad p(x,k\mid y)\propto p(y\mid x,k)\,p(x)\,p(k).

yy 是模糊觀測,p(x)p(x)、p(k)p(k) 是分別訓練的影像與核先驗。兩者獨立是觀察 yy 之前的建模假設;後驗中由概似把它們耦合。原文也把亂流建模為先傾斜/扭曲、再模糊,並為額外算子場建立先驗。

並行跑先驗,再共同修正。 時間 tt 的影像、核分數網路產生乾淨估計 x^0(xt)\widehat x_0(x_t)、k^0(kt)\widehat k_0(k_t),共同資料項使用

rt=y−k^0∗x^0.r_t=y-\widehat k_0*\widehat x_0.

xt,ktx_t,k_t 是各自帶雜訊的擴散狀態,rtr_t 為量測殘差。影像引導對影像估計反傳,把核狀態視為另一變數;核引導則對核估計反傳。兩組梯度都依賴當前兩個未知量的估計。

實作分別提出反向擴散狀態,再做概似修正。原文的實務演算法使用殘差範數與固定係數,而非要求每一步都套用精確的高斯平方殘差係數。數值選擇與機率動機應分開理解。

限制模糊核並加入有用正規性。 乾淨核估計投影到非負、總和一的集合,修正常見尺度歧義並排除負點擴散函數,但不能證明影像/核解唯一。稀疏項進一步偏好合適運動核;其權重是額外假設,並非所有真實光學模糊都共享的性質。

算子先驗的訓練是方法的一部分。BlindDPS 不是完全不改變的無條件影像模型,在單次推論中就能自動學會任何未知物理退化。

讀圖與選用判斷。 追蹤影像與模糊核兩個乾淨估計如何共同形成預測量測。若更新影像時忽略目前的核,或更新核時假設影像已知,就會失去這個方法的耦合關係。殘差小並不足以辨識正確組合。應同時檢查估計核與修復影像,並區分「訓練算子先驗」和「測試時估計算子」。相較已知算子的修復,這個方法增加不確定性與運算;額外先驗是否有用,取決於它是否能代表實際遇到的退化。

結果

原表 1 的 FFHQ 256×256256\times256 盲運動去模糊為 FID 29.49/LPIPS 0.281/PSNR 22.24 dB。表 2 另外評估核估計,而非只看影像;FFHQ 運動核的正規化卷積分數為 0.955。

表 3 呈現亂流的感知/失真取捨:FFHQ 的 BlindDPS 為 FID 27.35/PSNR 24.49 dB,監督式 TSR-WGAN 比較列則為 FID 58.30/PSNR 26.29 dB。感知分布分數改善,不代表 PSNR 較高或保證找回真實場景。

消融實驗

圖 7 用均勻先驗替換擴散核先驗,讓核估計只依概似引導。較差的核與影像結果支持學習算子先驗,但未分離所有可能替代正規化方法。

第 4.3 節與補充資料的稀疏研究改變核正規化。省略此項較差,尤其對運動模糊;一段非零強度範圍在測試設定有效。結果支持結合學習與手設先驗,並未建立所有未知相機皆適用的稀疏權重。

結論

BlindDPS 把概似引導擴散擴展到多個未知變數,各有先驗,再以量測模型耦合其取樣。因此必須同時查看重建與算子估計。

限制與問題

不同影像/核配對可能解釋同一觀測。核或影像若超出先驗訓練分布,可能被推向合理卻錯誤的解。空間變動模糊、飽和與未建模感測效應,可能違反前向假設。每增加未知量,都需要建模與計算;可以再加入一個擴散過程,不代表成本或可識別性免費獲得。

參考文獻

  1. Y. Blau and T. Michaeli, “The Perception-Distortion Tradeoff,” in Proc. IEEE/CVF CVPR, 2018, pp. 6228–6237 (extended version on arXiv). doi · arXiv
  2. C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang and W. Shi, “Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network,” in Proc. IEEE CVPR, 2017, pp. 105–114. doi · arXiv
  3. J. Johnson, A. Alahi and L. Fei-Fei, “Perceptual Losses for Real-Time Style Transfer and Super-Resolution,” in Proc. ECCV, 2016, pp. 694–711. doi · arXiv
  4. X. Wang, K. Yu, S. Wu, J. Gu, Y. Liu, C. Dong, Y. Qiao and C. C. Loy, “ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks,” in Proc. ECCV Workshops, 2018 (LNCS, 2019), pp. 63–79. doi · arXiv
  5. X. Wang, L. Xie, C. Dong and Y. Shan, “Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data,” in Proc. IEEE/CVF ICCV Workshops, 2021, pp. 1905–1914. doi · arXiv
  6. T. Karras, S. Laine and T. Aila, “A Style-Based Generator Architecture for Generative Adversarial Networks,” in Proc. IEEE/CVF CVPR, 2019, pp. 4396–4405. doi · arXiv
  7. S. Menon, A. Damian, S. Hu, N. Ravi and C. Rudin, “PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Models,” in Proc. IEEE/CVF CVPR, 2020, pp. 2434–2442. doi · arXiv
  8. K. C. K. Chan, X. Wang, X. Xu, J. Gu and C. C. Loy, “GLEAN: Generative Latent Bank for Large-Factor Image Super-Resolution,” in Proc. IEEE/CVF CVPR, 2021, pp. 14240–14249. doi · arXiv
  9. X. Wang, Y. Li, H. Zhang and Y. Shan, “Towards Real-World Blind Face Restoration with Generative Facial Prior,” in Proc. IEEE/CVF CVPR, 2021, pp. 9164–9174. doi · arXiv
  10. S. Zhou, K. C. K. Chan, C. Li and C. C. Loy, “Towards Robust Blind Face Restoration with Codebook Lookup Transformer,” in Proc. NeurIPS, vol. 35, 2022, pp. 30599–30611. doi · arXiv
  11. J. Ho, A. Jain and P. Abbeel, “Denoising Diffusion Probabilistic Models,” in Proc. NeurIPS, 2020. proceedings · arXiv
  12. Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon and B. Poole, “Score-Based Generative Modeling through Stochastic Differential Equations,” in Proc. ICLR, 2021. arXiv
  13. B. Efron, “Tweedie’s Formula and Selection Bias,” Journal of the American Statistical Association, vol. 106, no. 496, pp. 1602–1614, 2011. doi
  14. C. Saharia, J. Ho, W. Chan, T. Salimans, D. J. Fleet and M. Norouzi, “Image Super-Resolution via Iterative Refinement,” IEEE TPAMI, 2022 (early access). doi · arXiv
  15. C. Saharia, W. Chan, H. Chang, C. Lee, J. Ho, T. Salimans, D. Fleet and M. Norouzi, “Palette: Image-to-Image Diffusion Models,” in Proc. ACM SIGGRAPH Conference, 2022, pp. 1–10. doi · arXiv
  16. Z. Yue, J. Wang and C. C. Loy, “ResShift: Efficient Diffusion Model for Image Super-resolution by Residual Shifting,” in Proc. NeurIPS, vol. 36, 2023, pp. 13294–13307. doi · arXiv
  17. H. Chung, J. Kim, M. T. McCann, M. L. Klasky and J. C. Ye, “Diffusion Posterior Sampling for General Noisy Inverse Problems,” in Proc. ICLR, 2023. arXiv
  18. B. Kawar, M. Elad, S. Ermon and J. Song, “Denoising Diffusion Restoration Models,” in Proc. NeurIPS, vol. 35, 2022, pp. 23593–23606. doi · arXiv
  19. Y. Wang, J. Yu and J. Zhang, “Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Model,” in Proc. ICLR, 2023. arXiv
  20. J. Song, A. Vahdat, M. Mardani and J. Kautz, “Pseudoinverse-Guided Diffusion Models for Inverse Problems,” in Proc. ICLR, 2023. ICLR page
  21. Y. Zhu, K. Zhang, J. Liang, J. Cao, B. Wen, R. Timofte and L. Van Gool, “Denoising Diffusion Models for Plug-and-Play Image Restoration,” in Proc. IEEE/CVF CVPR Workshops, 2023, pp. 1219–1229. doi · arXiv
  22. R. Rombach, A. Blattmann, D. Lorenz, P. Esser and B. Ommer, “High-Resolution Image Synthesis with Latent Diffusion Models,” in Proc. IEEE/CVF CVPR, 2022, pp. 10674–10685. doi · arXiv
  23. L. Zhang, A. Rao and M. Agrawala, “Adding Conditional Control to Text-to-Image Diffusion Models,” in Proc. IEEE/CVF ICCV, 2023, pp. 3813–3824. doi · arXiv
  24. J. Wang, Z. Yue, S. Zhou, K. C. K. Chan and C. C. Loy, “Exploiting Diffusion Prior for Real-World Image Super-Resolution,” International Journal of Computer Vision, vol. 132, no. 12, pp. 5929–5949, 2024. doi · arXiv
  25. X. Lin, J. He, Z. Chen, Z. Lyu, B. Dai, F. Yu, Y. Qiao, W. Ouyang and C. Dong, “DiffBIR: Toward Blind Image Restoration with Generative Diffusion Prior,” in Proc. ECCV, 2024 (LNCS), pp. 430–448. doi · arXiv
  26. F. Yu, J. Gu, Z. Li, J. Hu, X. Kong, X. Wang, J. He, Y. Qiao and C. Dong, “Scaling Up to Excellence: Practicing Model Scaling for Photo-Realistic Image Restoration In the Wild,” in Proc. IEEE/CVF CVPR, 2024, pp. 25669–25680. doi · arXiv
  27. Y. Wang, W. Yang, X. Chen, Y. Wang, L. Guo, L.-P. Chau, Z. Liu, Y. Qiao, A. C. Kot and B. Wen, “SinSR: Diffusion-Based Image Super-Resolution in a Single Step,” in Proc. IEEE/CVF CVPR, 2024, pp. 25796–25805. doi · arXiv
  28. R. Wu, L. Sun, Z. Ma and L. Zhang, “One-Step Effective Diffusion Network for Real-World Image Super-Resolution,” in Proc. NeurIPS, vol. 37, 2024, pp. 92529–92553. doi · arXiv
  29. Z. Yue, K. Liao and C. C. Loy, “Arbitrary-steps Image Super-resolution via Diffusion Inversion,” in Proc. IEEE/CVF CVPR, 2025, pp. 23153–23163. doi · arXiv
  30. L. Dong, Q. Fan, Y. Guo, Z. Wang, Q. Zhang, J. Chen, Y. Luo and C. Zou, “TSD-SR: One-Step Diffusion with Target Score Distillation for Real-World Image Super-Resolution,” in Proc. IEEE/CVF CVPR, 2025, pp. 23174–23184. doi · arXiv
  31. Y. Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel and M. Le, “Flow Matching for Generative Modeling,” in Proc. ICLR, 2023. arXiv
  32. X. Liu, C. Gong and Q. Liu, “Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow,” in Proc. ICLR, 2023. arXiv
  33. G. Ohayon, T. Michaeli and M. Elad, “Posterior-Mean Rectified Flow: Towards Minimum MSE Photo-Realistic Image Restoration,” in Proc. ICLR, 2025. arXiv
  34. S. Martin, A. Gagneux, P. Hagemann and G. Steidl, “PnP-Flow: Plug-and-Play Image Restoration with Flow Matching,” in Proc. ICLR, 2025. arXiv
  35. S. Bhadra, V. A. Kelkar, F. J. Brooks and M. A. Anastasio, “On Hallucinations in Tomographic Image Reconstruction,” IEEE Transactions on Medical Imaging, vol. 40, no. 11, pp. 3249–3260, 2021. doi · arXiv
  36. W. Ren, R. Goyal, Z. Hu, T. T. Aumentado-Armstrong, I. Mohomed and A. Levinshtein, “Hallucination Score: Towards Mitigating Hallucinations in Generative Image Super-Resolution,” arXiv:2507.14367, 2025. arXiv
  37. A. N. Angelopoulos, A. P. Kohli, S. Bates, M. I. Jordan, J. Malik, T. Alshaabi, S. Upadhyayula and Y. Romano, “Image-to-Image Regression with Distribution-Free Uncertainty Quantification and Applications in Imaging,” in Proc. ICML, PMLR 162, 2022. proceedings · arXiv
  38. R. Zhang, P. Isola, A. A. Efros, E. Shechtman and O. Wang, “The Unreasonable Effectiveness of Deep Features as a Perceptual Metric,” in Proc. IEEE/CVF CVPR, 2018, pp. 586–595. doi · arXiv
  39. Z. Wang, A. C. Bovik, H. R. Sheikh and E. P. Simoncelli, “Image Quality Assessment: From Error Visibility to Structural Similarity,” IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600–612, 2004. doi
  40. M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler and S. Hochreiter, “GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium,” in Proc. NeurIPS (NIPS), 2017. arXiv
  41. A. Mittal, R. Soundararajan and A. C. Bovik, “Making a ‘Completely Blind’ Image Quality Analyzer,” IEEE Signal Processing Letters, vol. 20, no. 3, pp. 209–212, 2013. doi
  42. J. Ke, Q. Wang, Y. Wang, P. Milanfar and F. Yang, “MUSIQ: Multi-scale Image Quality Transformer,” in Proc. IEEE/CVF ICCV, 2021, pp. 5128–5137. doi · arXiv
  43. J. Wang, K. C. K. Chan and C. C. Loy, “Exploring CLIP for Assessing the Look and Feel of Images,” in Proc. AAAI, vol. 37, no. 2, 2023, pp. 2555–2563. doi · arXiv
  44. X. Li, Y. Ren, X. Jin, C. Lan, X. Wang, W. Zeng, X. Wang and Z. Chen, “Diffusion Models for Image Restoration and Enhancement: A Comprehensive Survey,” International Journal of Computer Vision, vol. 133, no. 11, pp. 8078–8108, 2025. doi · arXiv
  45. G. Daras, H. Chung, C.-H. Lai, Y. Mitsufuji, J. C. Ye, P. Milanfar, A. G. Dimakis and M. Delbracio, “A Survey on Diffusion Models for Inverse Problems,” arXiv:2410.00083, 2024. arXiv
  46. B. B. Moser, A. S. Shanbhag, F. Raue, S. Frolov, S. Palacio and A. Dengel, “Diffusion Models, Image Super-Resolution, and Everything: A Survey,” IEEE Transactions on Neural Networks and Learning Systems, vol. 36, no. 7, pp. 11793–11813, 2025. doi · arXiv
  47. W. Xia, Y. Zhang, Y. Yang, J.-H. Xue, B. Zhou and M.-H. Yang, “GAN Inversion: A Survey,” IEEE TPAMI, vol. 45, no. 3, pp. 3121–3138, 2023. doi · arXiv
  48. Y. Song and S. Ermon, “Generative Modeling by Estimating Gradients of the Data Distribution,” in Proc. NeurIPS, 2019. arXiv
  49. H. Chung, J. Kim, S. Kim and J. C. Ye, “Parallel Diffusion Models of Operator and Image for Blind Inverse Problems,” in Proc. IEEE/CVF CVPR, 2023, pp. 6059–6069. paper