完整搜尋頁面 →

DenseFuse:學習重建,在特徵空間選擇融合規則

選讀・DLCV 10・影像融合:互補來源與有用輸出 的論文導讀(第 4/11 篇)。引用編號對應本頁底部的參考文獻。

導覽:本頁與本系列: 導覽:本頁與本系列

本頁內容

補充資料 · 6

  1. 散焦擴散效應
  2. 建構與還原拉普拉斯金字塔
  3. 稀疏表示融合
  4. 互資訊作為融合指標
  5. 代表性融合網路的訓練訊號
  6. VIF 與來源參考的融合分數

論文導讀 · 11

  1. 曝光融合(Mertens、Kautz 與 Van Reeth)
  2. VIFB:可見光與紅外線影像融合基準
  3. Liu 等人的紅外線與可見光融合綜述(TPAMI 2025)
  4. DenseFuse:學習重建,在特徵空間選擇融合規則
  5. DDFM:以概似修正擴散先驗
  6. FusionGAN:熱強度與對抗式可見光先驗
  7. IFCNN:支援可變輸入數的監督式融合
  8. U2Fusion:跨融合任務共用一個無監督模型
  9. SwinFusion:來源內與來源間的注意力
  10. SeAFusion:讓語意分割影響融合影像
  11. TarDAL:同時為外觀與物件偵測融合

電腦視覺深度學習

  1. 02 · 深度影像分類
  2. 03 · 物件偵測:從 R-CNN 到 DETR
  3. 09 · 深度影像復原:低階視覺
查看完整課程樹 →

Li 與 Wu 的 TIP 2019 論文把影像表示學習與來源組合分開 [30]。編碼器、解碼器先學單張影像重建,測試時才在兩者之間插入固定融合規則。

挑戰

傳統融合使用設計好的轉換與活動度。學習式編碼器能提供更豐富表示,但紅外線—可見光配對沒有唯一融合真值。DenseFuse 先解有答案的問題:網路能否保留有用細節,重建自己的輸入?

訓練目標是一般影像,而非正確的跨模態融合。良好重建讓表示可用,卻不能單獨決定最佳融合方式;融合規則仍是明確的設計選擇。

方法

學得的編碼器、解碼器包圍另外選定的特徵融合規則;學會重建不等於學會推論融合規則。
原論文圖 1 — 學得的編碼器、解碼器包圍另外選定的特徵融合規則;學會重建不等於學會推論融合規則。

圖片來源: [30], arXiv:1804.08361v9, PDF 第 2 頁. 原始 PDF.

編碼器。 3×33\times3 卷積將單通道轉為 16 通道,再由三層密集區塊各增加 16 通道。各層串接先前特徵,輸入依序為 16、32、48 通道,合併輸出為 64 通道 [30]。步幅一與反射填補維持尺寸,因此 H×WH\times W 影像對應 H×W×64H\times W\times64。

密集連接讓中間特徵持續可用,不必把所有細節擠進最後 16 通道。但這不證明可逆,也不保證每個顯著物件都保留;須用重建與融合實驗檢驗。

解碼器。 四個 3×33\times3 卷積把 64 通道依序轉為 64、32、16、1。中間層有整流啟動,最後一層重建影像。單圖訓練與融合特徵推論共用這個解碼器。

重建訓練。 令 Eθ,DθE_\theta,D_\theta 為編碼器與解碼器,輸出為 O=Dθ(Eθ(I))O=D_\theta(E_\theta(I))。手稿寫為

L=Lp+λ(1−SSIM⁡(O,I)),Lp=∥O−I∥2.\mathcal L=\mathcal L_p+\lambda\bigl(1-\operatorname{SSIM}(O,I)\bigr), \qquad \mathcal L_p=\|O-I\|_2.

II 是灰階訓練圖,OO 是重建,λ\lambda 是結構損失權重,SSIM 為結構相似性 [16]。手稿的像素項寫作歐氏範數;移用權重前須核對實作的縮減與尺度。論文測試 λ=1,10,100,1000\lambda=1,10,100,1000 [30],大權重反映不同損失尺度,而非四種物理品質定義。

訓練把 COCO 一般影像轉灰階並縮為 256×256256\times256;所述切分約有 79,000 張訓練、1,000 張驗證影像 [30]。學完後,融合時固定編解碼器。

相加規則。 各來源編碼為 ϕim(x,y)\phi_i^m(x,y),ii 索引來源,mm 索引通道,(x,y)(x,y) 是位置。最簡單融合為 fm=∑iϕimf^m=\sum_i\phi_i^m,是總和,不是平均,因此幅度可能不同於單圖重建時的特徵分布。

活動度加權。 先算 Ci(x,y)=∑m∣ϕim(x,y)∣C_i(x,y)=\sum_m|\phi_i^m(x,y)|,以 3×33\times3 鄰域平均得 C^i\hat C_i,跨來源正規化再融合:

wi(x,y)=C^i(x,y)∑jC^j(x,y),fm(x,y)=∑iwi(x,y)ϕim(x,y).w_i(x,y)=\frac{\hat C_i(x,y)}{\sum_j\hat C_j(x,y)},\qquad f^m(x,y)=\sum_i w_i(x,y)\phi_i^m(x,y).

同位置各通道共用 wiw_i。活動度全零時須定義退回規則,例如等權重。原文雖稱「soft-max」,顯示的公式是除以活動度總和,不是指數 softmax。鄰域平均則鼓勵局部一致權重。

最後解碼 ff。全卷積影像路徑可接受超過訓練裁切的尺寸,但仍受實作、記憶體限制,且假設來源已配準。

結果

手稿以 20 組紅外線—可見光配對評估,表 II 有七種指標 [30]。相加且 λ=100\lambda=100 時,QAB/F=0.44190Q^{AB/F}=0.44190、SCD 1.84854、來源平均 SSIM 0.73186。這是小型測試集的各項結果,並非所有指標皆最佳。

表中 DeepFuse 的 QAB/F=0.43989Q^{AB/F}=0.43989、來源平均 SSIM 0.72949,另有方法的熵更高 [30]。因此本章分開看資訊、邊緣與結構量;SSIM 較好不等於偵測或熱保真較好。

消融實驗

表 II 同時改融合策略與結構權重。相加版本的來源平均 SSIM 隨 λ=1,10,100,1000\lambda=1,10,100,1000,依序為 0.73159、0.73246、0.73186、0.73259 [30];但 QAB/FQ^{AB/F} 從 λ=100\lambda=100 的 0.44190 降為 1000 的 0.43831。結構權重增加不會改善每個融合分數。

相加與活動度加權的比較,固定表示學習框架而改推論規則,檢查人工合併規則的貢獻。重建損失曲線另顯示權重如何影響訓練,但不是逐個密集連接的完整架構消融。

結論

DenseFuse 是搭配明確融合規則的學習式表示,連接傳統轉換融合與端到端融合:分解、重建由資料學習,活動度與組合仍可直接檢查。

限制與問題

重建不會教跨模態相容性。熱輸入與合併特徵可能偏離訓練灰階自然影像;活動度可能偏好雜訊,相加可能改變尺度,兩者都不知道下游任務重視哪個物件。

小型基準與來源分數也限制泛化宣稱。U2Fusion 直接以來源相依損失訓練輸出,後續注意力則學更豐富互動;它們改變融合決策,而非只加深編碼器。

參考文獻

  1. Y. Liu, X. Chen, Z. Wang, Z. J. Wang, R. K. Ward and X. Wang, “Deep learning for pixel-level image fusion: Recent advances and future prospects,” Information Fusion, vol. 42, pp. 158–173, 2018. doi
  2. J. Liu, G. Wu, Z. Liu, D. Wang, Z. Jiang, L. Ma, W. Zhong, X. Fan and R. Liu, “Infrared and visible image fusion: From data compatibility to task adaption,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 47, no. 4, pp. 2349–2369, 2025. doi arXiv
  3. J. Ma, Y. Ma and C. Li, “Infrared and visible image fusion methods and applications: A survey,” Information Fusion, vol. 45, pp. 153–178, 2019. doi
  4. P. E. Debevec and J. Malik, “Recovering high dynamic range radiance maps from photographs,” in Proc. SIGGRAPH, 1997, pp. 369–378. doi
  5. T. Mertens, J. Kautz and F. Van Reeth, “Exposure fusion: A simple and practical alternative to high dynamic range photography,” Computer Graphics Forum, vol. 28, no. 1, pp. 161–171, 2009. doi
  6. S. Xu, X. Wei, C. Zhang, J. Liu and J. Zhang, “MFFW: A new dataset for multi-focus image fusion,” arXiv:2002.04780, 2020. arXiv
  7. X. Zhang, “Multi-focus image fusion: A benchmark,” arXiv:2005.01116, 2020. arXiv
  8. P. J. Burt and E. H. Adelson, “The Laplacian pyramid as a compact image code,” IEEE Transactions on Communications, vol. 31, no. 4, pp. 532–540, 1983. doi
  9. P. J. Burt and R. J. Kolczynski, “Enhanced image capture through fusion,” in Proc. 4th International Conference on Computer Vision (ICCV), 1993, pp. 173–182. doi
  10. P. J. Burt and E. H. Adelson, “A multiresolution spline with application to image mosaics,” ACM Transactions on Graphics, vol. 2, no. 4, pp. 217–236, 1983. doi
  11. H. Li, B. S. Manjunath and S. K. Mitra, “Multisensor image fusion using the wavelet transform,” Graphical Models and Image Processing, vol. 57, no. 3, pp. 235–245, 1995. doi
  12. S. Li, X. Kang and J. Hu, “Image fusion with guided filtering,” IEEE Transactions on Image Processing, vol. 22, no. 7, pp. 2864–2875, 2013. doi
  13. K. He, J. Sun and X. Tang, “Guided image filtering,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, no. 6, pp. 1397–1409, 2013. doi
  14. B. Yang and S. Li, “Multifocus image fusion and restoration with sparse representation,” IEEE Transactions on Instrumentation and Measurement, vol. 59, no. 4, pp. 884–892, 2010. doi
  15. M. Nejati, S. Samavi and S. Shirani, “Multi-focus image fusion using dictionary-based sparse representation,” Information Fusion, vol. 25, pp. 72–84, 2015. doi
  16. Z. Wang, A. C. Bovik, H. R. Sheikh and E. P. Simoncelli, “Image quality assessment: From error visibility to structural similarity,” IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600–612, 2004. doi
  17. X. Zhang, P. Ye and G. Xiao, “VIFB: A visible and infrared image fusion benchmark,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2020. CVF arXiv
  18. X. Zhang, “Benchmarking and comparing multi-exposure image fusion algorithms,” Information Fusion, vol. 74, pp. 111–131, 2021. doi arXiv
  19. A. M. Eskicioglu and P. S. Fisher, “Image quality measures and their performance,” IEEE Transactions on Communications, vol. 43, no. 12, pp. 2959–2965, 1995. doi
  20. G. Qu, D. Zhang and P. Yan, “Information measure for performance of image fusion,” Electronics Letters, vol. 38, no. 7, pp. 313–315, 2002. doi
  21. M. Hossny, S. Nahavandi and D. Creighton, “Comments on ‘Information measure for performance of image fusion’,” Electronics Letters, vol. 44, no. 18, pp. 1066–1067, 2008. doi
  22. C. S. Xydeas and V. Petrović, “Objective image fusion performance measure,” Electronics Letters, vol. 36, no. 4, pp. 308–309, 2000. doi
  23. G. Piella and H. Heijmans, “A new quality metric for image fusion,” in Proc. IEEE International Conference on Image Processing (ICIP), 2003, vol. 3, pp. III-173–176. doi
  24. Y. Han, Y. Cai, Y. Cao and X. Xu, “A new image fusion performance metric based on visual information fidelity,” Information Fusion, vol. 14, no. 2, pp. 127–135, 2013. doi
  25. K. Ma, K. Zeng and Z. Wang, “Perceptual quality assessment for multi-exposure image fusion,” IEEE Transactions on Image Processing, vol. 24, no. 11, pp. 3345–3356, 2015. doi
  26. Z. Liu, E. Blasch, Z. Xue, J. Zhao, R. Laganiere and W. Wu, “Objective assessment of multiresolution image fusion algorithms for context enhancement in night vision: A comparative study,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 34, no. 1, pp. 94–109, 2012. doi
  27. J. Liu, X. Fan, Z. Huang, G. Wu, R. Liu, W. Zhong and Z. Luo, “Target-aware dual adversarial learning and a multi-scenario multi-modality benchmark to fuse infrared and visible for object detection,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 5802–5811. CVF arXiv M3FD
  28. J. Liu, Z. Liu, G. Wu, L. Ma, R. Liu, W. Zhong, Z. Luo and X. Fan, “Multi-interactive feature learning and a full-time multi-modality benchmark for image fusion and segmentation,” in Proc. IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 8115–8124. CVF arXiv
  29. Y. Zhang, Y. Liu, P. Sun, H. Yan, X. Zhao and L. Zhang, “IFCNN: A general image fusion framework based on convolutional neural network,” Information Fusion, vol. 54, pp. 99–118, 2020. doi
  30. H. Li and X.-J. Wu, “DenseFuse: A fusion approach to infrared and visible images,” IEEE Transactions on Image Processing, vol. 28, no. 5, pp. 2614–2623, 2019. doi arXiv
  31. K. R. Prabhakar, V. S. Srikar and R. V. Babu, “DeepFuse: A deep unsupervised approach for exposure fusion with extreme exposure image pairs,” in Proc. IEEE International Conference on Computer Vision (ICCV), 2017, pp. 4724–4732. doi CVF
  32. H. Xu, J. Ma, J. Jiang, X. Guo and H. Ling, “U2Fusion: A unified unsupervised image fusion network,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 1, pp. 502–518, 2022. doi RoadScene
  33. L. Tang, J. Yuan and J. Ma, “Image fusion in the loop of high-level vision tasks: A semantic-aware real-time infrared and visible image fusion network,” Information Fusion, vol. 82, pp. 28–42, 2022. doi
  34. J. Ma, L. Tang, F. Fan, J. Huang, X. Mei and Y. Ma, “SwinFusion: Cross-domain long-range learning for general image fusion via Swin Transformer,” IEEE/CAA Journal of Automatica Sinica, vol. 9, no. 7, pp. 1200–1217, 2022. doi
  35. X. Zhang and Y. Demiris, “Visible and infrared image fusion using deep learning,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 8, pp. 10535–10554, 2023. doi
  36. Y. Liu, X. Chen, H. Peng and Z. Wang, “Multi-focus image fusion with a deep convolutional neural network,” Information Fusion, vol. 36, pp. 191–207, 2017. doi
  37. K. Ma, Z. Duanmu, H. Zhu, Y. Fang and Z. Wang, “Deep guided learning for fast multi-exposure image fusion,” IEEE Transactions on Image Processing, vol. 29, pp. 2808–2819, 2020. doi
  38. Z. Zhao, H. Bai, J. Zhang, Y. Zhang, S. Xu, Z. Lin, R. Timofte and L. Van Gool, “CDDFuse: Correlation-driven dual-branch feature decomposition for multi-modality image fusion,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 5906–5916. CVF arXiv
  39. J. Ma, W. Yu, P. Liang, C. Li and J. Jiang, “FusionGAN: A generative adversarial network for infrared and visible image fusion,” Information Fusion, vol. 48, pp. 11–26, 2019. doi · public article transcription
  40. H. Bai, J. Zhang, Z. Zhao, Y. Wu, L. Deng, Y. Cui, T. Feng and S. Xu, “Task-driven image fusion with learnable fusion loss,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025, pp. 7457–7468. CVF arXiv
  41. Z. Zhao, H. Bai, Y. Zhu, J. Zhang, S. Xu, Y. Zhang, K. Zhang, D. Meng, R. Timofte and L. Van Gool, “DDFM: Denoising diffusion model for multi-modality image fusion,” in Proc. IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 8082–8093. CVF arXiv
  42. X. Yi, H. Xu, H. Zhang, L. Tang and J. Ma, “Text-IF: Leveraging semantic text guidance for degradation-aware and interactive image fusion,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 27026–27035. CVF arXiv
  43. D. Wang, J. Liu, X. Fan and R. Liu, “Unsupervised misaligned infrared and visible image fusion via cross-modality image generation and registration,” in Proc. International Joint Conference on Artificial Intelligence (IJCAI), 2022, pp. 3508–3515. doi arXiv
  44. L. Tang, Y. Deng, Y. Ma, J. Huang and J. Ma, “SuperFusion: A versatile image registration and fusion network with semantic awareness,” IEEE/CAA Journal of Automatica Sinica, vol. 9, no. 12, pp. 2121–2137, 2022. doi
  45. A. Toet, “The TNO multiband image data collection,” Data in Brief, vol. 15, pp. 249–251, 2017. doi
  46. L. Tang, J. Yuan, H. Zhang, X. Jiang and J. Ma, “PIAFusion: A progressive infrared and visible image fusion network based on illumination aware,” Information Fusion, vol. 83–84, pp. 79–92, 2022. doi MSRS
  47. Q. Ha, K. Watanabe, T. Karasawa, Y. Ushiku and T. Harada, “MFNet: Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes,” in Proc. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2017, pp. 5108–5115. doi
  48. X. Jia, C. Zhu, M. Li, W. Tang and W. Zhou, “LLVIP: A visible-infrared paired dataset for low-light vision,” in Proc. IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, 2021, pp. 3496–3504. CVF arXiv
  49. J. Cai, S. Gu and L. Zhang, “Learning a deep single image contrast enhancer from multi-exposure images,” IEEE Transactions on Image Processing, vol. 27, no. 4, pp. 2049–2062, 2018. doi
  50. X. Zhang, “Deep learning-based multi-focus image fusion: A survey and a comparative study,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 9, 2022. doi
  51. F. Xu, J. Liu, Y. Song, H. Sun and X. Wang, “Multi-exposure image fusion techniques: A comprehensive review,” Remote Sensing, vol. 14, no. 3, art. 771, 2022. doi
  52. Z. Zhao, H. Bai, J. Zhang, Y. Zhang, K. Zhang, S. Xu, D. Chen, R. Timofte and L. Van Gool, “Equivariant multi-modality image fusion,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 25912–25921. CVF arXiv
  53. L. Tang, Y. Wang, M. Gong, Z. Li, Y. Deng, X. Yi, C. Li, H. Xu, H. Zhang and J. Ma, “VideoFusion: A spatio-temporal collaborative network for multi-modal video fusion,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026, pp. 19559–19569. CVF
  54. N. Yokoya, C. Grohnfeldt and J. Chanussot, “Hyperspectral and multispectral data fusion: A comparative review of the recent literature,” IEEE Geoscience and Remote Sensing Magazine, 2017. doi
  55. H. R. Sheikh and A. C. Bovik, “Image information and visual quality,” IEEE Transactions on Image Processing, 2006. doi
  56. H. Xu et al., “U2Fusion,” official implementation accompanying the TPAMI paper, 2022. code
  57. K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” ICLR, 2015. arXiv
  58. J. Kirkpatrick et al., “Overcoming catastrophic forgetting in neural networks,” Proceedings of the National Academy of Sciences, 2017. doi
  59. Y. Zhang et al., “IFCNN,” official implementation and errata accompanying the Information Fusion paper, 2020. code
  60. K. He, X. Zhang, S. Ren and J. Sun, “Deep residual learning for image recognition,” CVPR, 2016. CVF
  61. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li and L. Fei-Fei, “ImageNet: A large-scale hierarchical image database,” CVPR, 2009. doi
  62. N. Silberman, D. Hoiem, P. Kohli and R. Fergus, “Indoor segmentation and support inference from RGBD images,” ECCV, 2012. project
  63. P. Dhariwal and A. Nichol, “Diffusion models beat GANs on image synthesis,” NeurIPS, 2021. arXiv
  64. J. Liu et al., “TarDAL,” official implementation accompanying the CVPR paper, 2022. code
  65. G. Jocher et al., “YOLOv5,” Ultralytics object-detection software, 2020 onward. code
  66. J. Ma et al., “FusionGAN,” official implementation accompanying the Information Fusion paper, 2019. code
  67. L. Tang, J. Yuan and J. Ma, “SeAFusion,” official implementation accompanying the Information Fusion paper, 2022. code
  68. T. Mertens, J. Kautz and F. Van Reeth, “Exposure Fusion,” in Proc. Pacific Graphics, 2007. paper · author project