DDFM:以概似修正擴散先驗
選讀・DLCV 10・影像融合:互補來源與有用輸出 的論文導讀(第 5/11 篇)。引用編號對應本頁底部的參考文獻。
導覽:本頁與本系列: 導覽:本頁與本系列
Zhao 等人的 ICCV 2023 論文結合預訓練自然影像擴散模型與明確來源保留模型 [41]。融合在取樣中進行,不用融合配對微調去雜訊器。
挑戰
無監督目標可定義保留什麼,卻未必有強影像先驗;無條件生成器則可能產生合理但無關來源的圖。DDFM 分工:擴散提供先驗,概似修正把每次去雜訊估計拉向兩個觀測。
困難在於,融合沒有已知的「唯一真實融合圖到兩感測器」物理退化算子。概似須經建模,不能當成已知感測定律;使用貝氏符號也不會消除這個假設。
方法

圖片來源: [41], Published paper PDF, PDF 第 6 頁. 原始 PDF.
無條件去雜訊。 時間 下,預訓練模型由 預測乾淨估計 。使用固定權重的 ImageNet 擴散模型 [63]、[61];它提供自然影像偏好,不直接決定保留哪個熱目標。
來源保留模型。 令 為紅外線、 為可見光、 為候選融合,起始目標為
決定來源相對偏好。設 、已知 ,可改寫為 。此變數替換將候選視為可見光修正量,並讓絕對值懲罰有機率表示。
潛在變異數。 類 Laplace 殘差可表示成高斯尺度混合:給定正的潛在變異數, 與 為高斯;變異數的指數先驗產生厚尾邊際懲罰。因此有效權重可隨空間變化,而非只有固定全域折衷。
期望最大化。 E 步依目前殘差估計潛在精確度統計;M 步解加權二次問題來更新修正。以對角精確度矩陣 、空間差分算子 示意為
加權紅外線殘差, 加權相對可見光的修正, 控制正規化。此式摘要條件二次結構,不是以新固定權重替換原文推論變數。末項平滑 ,鼓勵保留可見光紋理。原文雖稱全變分,顯示的平方梯度項其實為二次式,不同於常見絕對梯度 TV。
多個擴散步,每步一次 EM。 概似修正把 改為 ,擴散轉移再結合雜訊狀態與乾淨估計得到 ,沿反向軌跡重複。演算法 1 是每個擴散步一次 EM,不是每步求到 EM 完全收斂,也不是單步生成;外層仍反覆呼叫去雜訊器。
後驗分數推導透過條件生成近似來支持此分工,不能說成從已知物理後驗精確取樣。先驗、概似、有限更新與離散化都會影響結果。
結果
論文在 MSRS、M3FD、RoadScene、TNO 評估紅外線—可見光,醫學則用 50 組 Harvard 配對 [41]。表 1 的 MSRS 為 EN 6.88、MI 2.35、VIF 0.81、 0.58、SSIM 0.94;M3FD 的 MI 2.52、VIF 0.80,但 SSIM 0.95 低於 U2Fusion 的 0.98 [41],並非全項勝出。
表 3 依該工具箱得到醫學 VIF 0.76、 0.60 [41]。其中 SSIM 類值可能超過一,因採融合彙整慣例,不是一般單參考 SSIM。這些數值與合理外觀均不能證明診斷準確度。
消融實驗
表 2 在 RoadScene:移除擴散、只保留迭代 EM,得 VIF 0.60/ 0.42;移除修正正規化為 0.69/0.52;以固定 或 1 取代自適應取捨,為 0.66/0.49、0.66/0.47;完整模型為 0.75/0.65 [41]。
這支持先驗、正規化與自適應概似在該實驗的互補作用,但不代表分數改善就等於更準確的物理融合。EM 基準也受所述迭代預算限制,不是無限制最佳化的理想解。
結論
DDFM 讓預訓練影像先驗與來源保留模型反覆互動,展示不需為每個資料集另訓融合去雜訊器的生成式路線。
限制與問題
迭代取樣比單次前饋融合需要更多工作,結果受預訓練先驗、取樣器、權重與配準影響。強自然影像先驗可能改變不常見但有效的感測證據,來源指標也未必能揭露。
建模概似不是校準相機模型,貝氏寫法不會改變這點。應核對亮目標、解剖邊界是否由輸入支持;若涉及定量感測解讀,須同時保留原始模態。
參考文獻
- Y. Liu, X. Chen, Z. Wang, Z. J. Wang, R. K. Ward and X. Wang, “Deep learning for pixel-level image fusion: Recent advances and future prospects,” Information Fusion, vol. 42, pp. 158–173, 2018. doi
- J. Liu, G. Wu, Z. Liu, D. Wang, Z. Jiang, L. Ma, W. Zhong, X. Fan and R. Liu, “Infrared and visible image fusion: From data compatibility to task adaption,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 47, no. 4, pp. 2349–2369, 2025. doi arXiv
- J. Ma, Y. Ma and C. Li, “Infrared and visible image fusion methods and applications: A survey,” Information Fusion, vol. 45, pp. 153–178, 2019. doi
- P. E. Debevec and J. Malik, “Recovering high dynamic range radiance maps from photographs,” in Proc. SIGGRAPH, 1997, pp. 369–378. doi
- T. Mertens, J. Kautz and F. Van Reeth, “Exposure fusion: A simple and practical alternative to high dynamic range photography,” Computer Graphics Forum, vol. 28, no. 1, pp. 161–171, 2009. doi
- S. Xu, X. Wei, C. Zhang, J. Liu and J. Zhang, “MFFW: A new dataset for multi-focus image fusion,” arXiv:2002.04780, 2020. arXiv
- X. Zhang, “Multi-focus image fusion: A benchmark,” arXiv:2005.01116, 2020. arXiv
- P. J. Burt and E. H. Adelson, “The Laplacian pyramid as a compact image code,” IEEE Transactions on Communications, vol. 31, no. 4, pp. 532–540, 1983. doi
- P. J. Burt and R. J. Kolczynski, “Enhanced image capture through fusion,” in Proc. 4th International Conference on Computer Vision (ICCV), 1993, pp. 173–182. doi
- P. J. Burt and E. H. Adelson, “A multiresolution spline with application to image mosaics,” ACM Transactions on Graphics, vol. 2, no. 4, pp. 217–236, 1983. doi
- H. Li, B. S. Manjunath and S. K. Mitra, “Multisensor image fusion using the wavelet transform,” Graphical Models and Image Processing, vol. 57, no. 3, pp. 235–245, 1995. doi
- S. Li, X. Kang and J. Hu, “Image fusion with guided filtering,” IEEE Transactions on Image Processing, vol. 22, no. 7, pp. 2864–2875, 2013. doi
- K. He, J. Sun and X. Tang, “Guided image filtering,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, no. 6, pp. 1397–1409, 2013. doi
- B. Yang and S. Li, “Multifocus image fusion and restoration with sparse representation,” IEEE Transactions on Instrumentation and Measurement, vol. 59, no. 4, pp. 884–892, 2010. doi
- M. Nejati, S. Samavi and S. Shirani, “Multi-focus image fusion using dictionary-based sparse representation,” Information Fusion, vol. 25, pp. 72–84, 2015. doi
- Z. Wang, A. C. Bovik, H. R. Sheikh and E. P. Simoncelli, “Image quality assessment: From error visibility to structural similarity,” IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600–612, 2004. doi
- X. Zhang, P. Ye and G. Xiao, “VIFB: A visible and infrared image fusion benchmark,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2020. CVF arXiv
- X. Zhang, “Benchmarking and comparing multi-exposure image fusion algorithms,” Information Fusion, vol. 74, pp. 111–131, 2021. doi arXiv
- A. M. Eskicioglu and P. S. Fisher, “Image quality measures and their performance,” IEEE Transactions on Communications, vol. 43, no. 12, pp. 2959–2965, 1995. doi
- G. Qu, D. Zhang and P. Yan, “Information measure for performance of image fusion,” Electronics Letters, vol. 38, no. 7, pp. 313–315, 2002. doi
- M. Hossny, S. Nahavandi and D. Creighton, “Comments on ‘Information measure for performance of image fusion’,” Electronics Letters, vol. 44, no. 18, pp. 1066–1067, 2008. doi
- C. S. Xydeas and V. Petrović, “Objective image fusion performance measure,” Electronics Letters, vol. 36, no. 4, pp. 308–309, 2000. doi
- G. Piella and H. Heijmans, “A new quality metric for image fusion,” in Proc. IEEE International Conference on Image Processing (ICIP), 2003, vol. 3, pp. III-173–176. doi
- Y. Han, Y. Cai, Y. Cao and X. Xu, “A new image fusion performance metric based on visual information fidelity,” Information Fusion, vol. 14, no. 2, pp. 127–135, 2013. doi
- K. Ma, K. Zeng and Z. Wang, “Perceptual quality assessment for multi-exposure image fusion,” IEEE Transactions on Image Processing, vol. 24, no. 11, pp. 3345–3356, 2015. doi
- Z. Liu, E. Blasch, Z. Xue, J. Zhao, R. Laganiere and W. Wu, “Objective assessment of multiresolution image fusion algorithms for context enhancement in night vision: A comparative study,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 34, no. 1, pp. 94–109, 2012. doi
- J. Liu, X. Fan, Z. Huang, G. Wu, R. Liu, W. Zhong and Z. Luo, “Target-aware dual adversarial learning and a multi-scenario multi-modality benchmark to fuse infrared and visible for object detection,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 5802–5811. CVF arXiv M3FD
- J. Liu, Z. Liu, G. Wu, L. Ma, R. Liu, W. Zhong, Z. Luo and X. Fan, “Multi-interactive feature learning and a full-time multi-modality benchmark for image fusion and segmentation,” in Proc. IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 8115–8124. CVF arXiv
- Y. Zhang, Y. Liu, P. Sun, H. Yan, X. Zhao and L. Zhang, “IFCNN: A general image fusion framework based on convolutional neural network,” Information Fusion, vol. 54, pp. 99–118, 2020. doi
- H. Li and X.-J. Wu, “DenseFuse: A fusion approach to infrared and visible images,” IEEE Transactions on Image Processing, vol. 28, no. 5, pp. 2614–2623, 2019. doi arXiv
- K. R. Prabhakar, V. S. Srikar and R. V. Babu, “DeepFuse: A deep unsupervised approach for exposure fusion with extreme exposure image pairs,” in Proc. IEEE International Conference on Computer Vision (ICCV), 2017, pp. 4724–4732. doi CVF
- H. Xu, J. Ma, J. Jiang, X. Guo and H. Ling, “U2Fusion: A unified unsupervised image fusion network,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 1, pp. 502–518, 2022. doi RoadScene
- L. Tang, J. Yuan and J. Ma, “Image fusion in the loop of high-level vision tasks: A semantic-aware real-time infrared and visible image fusion network,” Information Fusion, vol. 82, pp. 28–42, 2022. doi
- J. Ma, L. Tang, F. Fan, J. Huang, X. Mei and Y. Ma, “SwinFusion: Cross-domain long-range learning for general image fusion via Swin Transformer,” IEEE/CAA Journal of Automatica Sinica, vol. 9, no. 7, pp. 1200–1217, 2022. doi
- X. Zhang and Y. Demiris, “Visible and infrared image fusion using deep learning,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 8, pp. 10535–10554, 2023. doi
- Y. Liu, X. Chen, H. Peng and Z. Wang, “Multi-focus image fusion with a deep convolutional neural network,” Information Fusion, vol. 36, pp. 191–207, 2017. doi
- K. Ma, Z. Duanmu, H. Zhu, Y. Fang and Z. Wang, “Deep guided learning for fast multi-exposure image fusion,” IEEE Transactions on Image Processing, vol. 29, pp. 2808–2819, 2020. doi
- Z. Zhao, H. Bai, J. Zhang, Y. Zhang, S. Xu, Z. Lin, R. Timofte and L. Van Gool, “CDDFuse: Correlation-driven dual-branch feature decomposition for multi-modality image fusion,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 5906–5916. CVF arXiv
- J. Ma, W. Yu, P. Liang, C. Li and J. Jiang, “FusionGAN: A generative adversarial network for infrared and visible image fusion,” Information Fusion, vol. 48, pp. 11–26, 2019. doi · public article transcription
- H. Bai, J. Zhang, Z. Zhao, Y. Wu, L. Deng, Y. Cui, T. Feng and S. Xu, “Task-driven image fusion with learnable fusion loss,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025, pp. 7457–7468. CVF arXiv
- Z. Zhao, H. Bai, Y. Zhu, J. Zhang, S. Xu, Y. Zhang, K. Zhang, D. Meng, R. Timofte and L. Van Gool, “DDFM: Denoising diffusion model for multi-modality image fusion,” in Proc. IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 8082–8093. CVF arXiv
- X. Yi, H. Xu, H. Zhang, L. Tang and J. Ma, “Text-IF: Leveraging semantic text guidance for degradation-aware and interactive image fusion,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 27026–27035. CVF arXiv
- D. Wang, J. Liu, X. Fan and R. Liu, “Unsupervised misaligned infrared and visible image fusion via cross-modality image generation and registration,” in Proc. International Joint Conference on Artificial Intelligence (IJCAI), 2022, pp. 3508–3515. doi arXiv
- L. Tang, Y. Deng, Y. Ma, J. Huang and J. Ma, “SuperFusion: A versatile image registration and fusion network with semantic awareness,” IEEE/CAA Journal of Automatica Sinica, vol. 9, no. 12, pp. 2121–2137, 2022. doi
- A. Toet, “The TNO multiband image data collection,” Data in Brief, vol. 15, pp. 249–251, 2017. doi
- L. Tang, J. Yuan, H. Zhang, X. Jiang and J. Ma, “PIAFusion: A progressive infrared and visible image fusion network based on illumination aware,” Information Fusion, vol. 83–84, pp. 79–92, 2022. doi MSRS
- Q. Ha, K. Watanabe, T. Karasawa, Y. Ushiku and T. Harada, “MFNet: Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes,” in Proc. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2017, pp. 5108–5115. doi
- X. Jia, C. Zhu, M. Li, W. Tang and W. Zhou, “LLVIP: A visible-infrared paired dataset for low-light vision,” in Proc. IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, 2021, pp. 3496–3504. CVF arXiv
- J. Cai, S. Gu and L. Zhang, “Learning a deep single image contrast enhancer from multi-exposure images,” IEEE Transactions on Image Processing, vol. 27, no. 4, pp. 2049–2062, 2018. doi
- X. Zhang, “Deep learning-based multi-focus image fusion: A survey and a comparative study,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 9, 2022. doi
- F. Xu, J. Liu, Y. Song, H. Sun and X. Wang, “Multi-exposure image fusion techniques: A comprehensive review,” Remote Sensing, vol. 14, no. 3, art. 771, 2022. doi
- Z. Zhao, H. Bai, J. Zhang, Y. Zhang, K. Zhang, S. Xu, D. Chen, R. Timofte and L. Van Gool, “Equivariant multi-modality image fusion,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 25912–25921. CVF arXiv
- L. Tang, Y. Wang, M. Gong, Z. Li, Y. Deng, X. Yi, C. Li, H. Xu, H. Zhang and J. Ma, “VideoFusion: A spatio-temporal collaborative network for multi-modal video fusion,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026, pp. 19559–19569. CVF
- N. Yokoya, C. Grohnfeldt and J. Chanussot, “Hyperspectral and multispectral data fusion: A comparative review of the recent literature,” IEEE Geoscience and Remote Sensing Magazine, 2017. doi
- H. R. Sheikh and A. C. Bovik, “Image information and visual quality,” IEEE Transactions on Image Processing, 2006. doi
- H. Xu et al., “U2Fusion,” official implementation accompanying the TPAMI paper, 2022. code
- K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” ICLR, 2015. arXiv
- J. Kirkpatrick et al., “Overcoming catastrophic forgetting in neural networks,” Proceedings of the National Academy of Sciences, 2017. doi
- Y. Zhang et al., “IFCNN,” official implementation and errata accompanying the Information Fusion paper, 2020. code
- K. He, X. Zhang, S. Ren and J. Sun, “Deep residual learning for image recognition,” CVPR, 2016. CVF
- J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li and L. Fei-Fei, “ImageNet: A large-scale hierarchical image database,” CVPR, 2009. doi
- N. Silberman, D. Hoiem, P. Kohli and R. Fergus, “Indoor segmentation and support inference from RGBD images,” ECCV, 2012. project
- P. Dhariwal and A. Nichol, “Diffusion models beat GANs on image synthesis,” NeurIPS, 2021. arXiv
- J. Liu et al., “TarDAL,” official implementation accompanying the CVPR paper, 2022. code
- G. Jocher et al., “YOLOv5,” Ultralytics object-detection software, 2020 onward. code
- J. Ma et al., “FusionGAN,” official implementation accompanying the Information Fusion paper, 2019. code
- L. Tang, J. Yuan and J. Ma, “SeAFusion,” official implementation accompanying the Information Fusion paper, 2022. code
- T. Mertens, J. Kautz and F. Van Reeth, “Exposure Fusion,” in Proc. Pacific Graphics, 2007. paper · author project