U2Fusion:跨融合任務共用一個無監督模型
選讀・DLCV 10・影像融合:互補來源與有用輸出 的論文導讀(第 8/11 篇)。引用編號對應本頁底部的參考文獻。
導覽:本頁與本系列: 導覽:本頁與本系列
Xu 等人的 TPAMI 2022 論文把紅外線—可見光、醫學、多曝光、多焦點視為相關資訊保留問題 [32],目標是沒有融合真值也能跨任務訓練同一組參數。以下依擴充 FusionDN 的期刊版說明。
挑戰
各任務偏好似乎不相容:熱目標應顯著,解剖結構要保留,欠曝光區即使梯度弱仍可能有內容,失焦區則應少於清晰區。一種固定活動度難以表達所有偏好。
有兩個問題:沒有融合真值,如何監督輸出?學下一任務時,如何保留舊能力?U2Fusion 以自適應相似損失解前者,以彈性權重鞏固(EWC)解後者;「統一」不只是同架構分別訓練。
方法

圖片來源: [32], Public manuscript PDF, PDF 第 4 頁. 原始 PDF.
1. 直接產圖。 沿通道串接兩張單通道影像,輸入 、輸出 。全卷積生成器採密集連接、反射填補、步幅一 卷積,無池化;早期特徵持續可供後層使用,也有利梯度傳遞。
圖 8 與官方程式指定初始 44 通道,再由五個密集卷積各加 44,最後將 264 依序減為 128、64、32、1,共十個卷積 [32]、[56]。正文「前七個密集層」與圖、程式不一致,此處依明確張量路徑。中間 LeakyReLU 斜率 0.2,最後 tanh;重現須配合輸入輸出尺度。
2. 固定預訓練特徵估資訊。 灰階複製成三通道送入 VGG-16 [57],取五次最大池化前的特徵。 區塊的邊長依序為 ,通道為 。淺層偏紋理,深層有較大脈絡;它定義訓練監督,不是第二生成器。
論文以特徵圖的拉普拉斯衡量活動度:
是來源區塊, 是第 階段第 通道, 是張量高、寬、通道數; 為空間拉普拉斯, 加總平方項。除以張量大小再平均各階段,得到來源區塊的純量活動度;它是設計的統計,不是 Shannon 資訊,也不是物理感測可靠度估計。
3. 轉成保留權重。 對來源區塊 計算
為任務特定尺度。權重非負、和為一,較小 放大來源差異。它是訓練區塊的來源權重,不是逐像素判斷哪個感測器正確的分割圖;跨區塊訓練後,鼓勵保留特徵活動度較高的來源。
4. 互補相似性。 對生成器 ,定義
為參數,期望平均配對, 索引兩來源,SSIM 比較結構與亮度統計 [16],MSE 為像素均方差, 控制取捨。期刊版用 [32];前者約束結構,後者約束強度,但都不提供唯一融合真值,高權重也不保證保留每個特徵。
期刊目標沒有保留 FusionDN 的梯度匹配損失;作者移除它以減少假邊緣,加入 MSE 約束亮度。不要把歷史討論提到的所有損失全加進 U2Fusion。
5. 以 EWC 依序學任務。 一般微調可能覆寫舊能力。EWC [58] 懲罰對舊任務重要的參數移動:
為舊任務完成值, 為目前值, 是重要性, 是鞏固強度。較大 受較強保護。論文以對角 Fisher 動機,使用舊任務在 的平均平方梯度近似;對此一般融合損失,應視為重要性近似,不是校準的機率不確定性。
丟棄舊資料前先估重要性,下一任務不必重播全部舊圖。完成多任務後,須依訓練程序累積重要性;只保留上一任務的無權重參數並不等同鞏固。
採用順序為多模態→多曝光→多焦點。多模態把可見光—紅外線與 PET–MRI 合為首任務。設定為 裁切、批次 18、學習率 、三任務各 3/2/2 epochs, 分別為 , [32]。尺度依賴特徵與強度慣例,不能直接移植到任意骨幹。
6. 彩色與多輸入。 彩圖轉 YCbCr,只學習融合亮度;色度另依偏離中性值的強度加權。對任一色度通道,原文為
是八位元慣例下的來源色度。兩者都中性時分母為零,實作可回傳 ;只有一來源有色彩時,使用其色度。這不是端到端學習色彩融合。
多來源則先融合兩張,再將中間結果與下一張融合。可接受多張,不表示二元生成器有結合律或不受順序影響;誤差與偏好可能累積。
推論做什麼。 只執行生成器與必要色彩轉換。VGG 活動度、自適應損失權重與 EWC 都屬訓練,不是額外推論網路,這是理解成本的關鍵。
結果
期刊評估六個集合:TNO、RoadScene、Harvard PET–MRI、訓練研究採用的多曝光集合、EMPA HDR、Lytro [32]。RoadScene 有 221 組對齊配對;定量可見光—紅外線表採 20 組 TNO、45 組 RoadScene,其他影像做定性展示。比較後續論文時須核對切分與前處理。
部分期刊值同時顯示優勢與限制:
| 設定 | 原文 U2Fusion 結果 | 解讀 |
|---|---|---|
| TNO,表 1 | CC ;SSIM ;PSNR | 依其來源比較定義 |
| RoadScene,表 1 | CC ;SCD | SCD 非最佳;FusionDN 為 |
| Harvard,表 2 | CC ;SSIM | 相關與結構保留,不是診斷驗證 |
| Lytro,表 4 | CC ;VIF | 小型多焦點基準的良好結果 |
接近 2 的 SSIM 是跨來源的融合彙整,不是上限 1 的單參考 SSIM。PSNR 也依來源比較,並沒有隱藏融合真值,不能與乾淨真值復原 PSNR 直接比。重現須用相同工具箱與尺度。
也不是每個指標勝出:表 3 EMPA HDR 的 PSNR 為 58.454,DeepFuse 為 58.518;表 4 FusionDN 的 VIF 也較高 [32]。應報完整脈絡,不只挑勝出的欄。
消融實驗
EWC。 圖 19–20 比較有無鞏固的序列訓練:沒有 EWC 時,後續任務會改變舊任務亮度與外觀,EWC 可減輕。重要性直方圖描述保護權重,這些定性與分布分析不是特定 AP 類增益表。
共用與獨立模型。 圖 21 比較分開訓練及統一模型,展示曝光學習幫助其他任務過曝區、多焦點訓練幫助曝光融合邊界的例子。這支持該示例的轉移,不保證每個新任務都改善全部舊任務。
自適應權重。 圖 22 把兩權重固定 0.5,相較自適應版本會丟細節、保留過曝區,多焦點邊界尤其模糊。原文提供影像,沒有完整數值表可推成普遍百分比增益。
訓練順序。 表 5 採用順序的 Lytro 為 CC 0.9723/平均梯度 0.0677;交換前兩任務為 0.9724/0.0700;先多焦點、最後曝光則 0.9803/0.0563 [32]。最後順序相關較高卻梯度較弱、邊界較軟。即使有 EWC,順序仍重要,指標也會分歧。
相較 FusionDN。 圖 24 同時更換特徵活動度、移除梯度損失、加入 MSE,並把醫學例子納入首任務。這些變化沒有完整因子實驗逐項量化,因此支持整體新配方,不能把全部改善歸給單一改動。
結論
U2Fusion 跨任務統一目標與參數,以固定特徵定義來源偏好,結構、強度訓練生成器,EWC 減輕序列遺忘。它把「多任務一模型」化為具體最佳化問題。
限制與問題
特徵活動度可能獎勵雜訊、偽影,且權重是每區塊純量。自然影像特徵不保證量對醫學或熱重要性;目標仍是設計偏好,不是物理融合真值。
假設已配準,色彩部分靠固定規則,多輸入結果受順序影響。EWC 減輕但不消除遺忘,順序表即為例。小資料與來源指標限制泛化。SwinFusion 改特徵互動,任務感知方法改問下游效益,但都仍需檢查對齊、保真與評估假設。
參考文獻
- Y. Liu, X. Chen, Z. Wang, Z. J. Wang, R. K. Ward and X. Wang, “Deep learning for pixel-level image fusion: Recent advances and future prospects,” Information Fusion, vol. 42, pp. 158–173, 2018. doi
- J. Liu, G. Wu, Z. Liu, D. Wang, Z. Jiang, L. Ma, W. Zhong, X. Fan and R. Liu, “Infrared and visible image fusion: From data compatibility to task adaption,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 47, no. 4, pp. 2349–2369, 2025. doi arXiv
- J. Ma, Y. Ma and C. Li, “Infrared and visible image fusion methods and applications: A survey,” Information Fusion, vol. 45, pp. 153–178, 2019. doi
- P. E. Debevec and J. Malik, “Recovering high dynamic range radiance maps from photographs,” in Proc. SIGGRAPH, 1997, pp. 369–378. doi
- T. Mertens, J. Kautz and F. Van Reeth, “Exposure fusion: A simple and practical alternative to high dynamic range photography,” Computer Graphics Forum, vol. 28, no. 1, pp. 161–171, 2009. doi
- S. Xu, X. Wei, C. Zhang, J. Liu and J. Zhang, “MFFW: A new dataset for multi-focus image fusion,” arXiv:2002.04780, 2020. arXiv
- X. Zhang, “Multi-focus image fusion: A benchmark,” arXiv:2005.01116, 2020. arXiv
- P. J. Burt and E. H. Adelson, “The Laplacian pyramid as a compact image code,” IEEE Transactions on Communications, vol. 31, no. 4, pp. 532–540, 1983. doi
- P. J. Burt and R. J. Kolczynski, “Enhanced image capture through fusion,” in Proc. 4th International Conference on Computer Vision (ICCV), 1993, pp. 173–182. doi
- P. J. Burt and E. H. Adelson, “A multiresolution spline with application to image mosaics,” ACM Transactions on Graphics, vol. 2, no. 4, pp. 217–236, 1983. doi
- H. Li, B. S. Manjunath and S. K. Mitra, “Multisensor image fusion using the wavelet transform,” Graphical Models and Image Processing, vol. 57, no. 3, pp. 235–245, 1995. doi
- S. Li, X. Kang and J. Hu, “Image fusion with guided filtering,” IEEE Transactions on Image Processing, vol. 22, no. 7, pp. 2864–2875, 2013. doi
- K. He, J. Sun and X. Tang, “Guided image filtering,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, no. 6, pp. 1397–1409, 2013. doi
- B. Yang and S. Li, “Multifocus image fusion and restoration with sparse representation,” IEEE Transactions on Instrumentation and Measurement, vol. 59, no. 4, pp. 884–892, 2010. doi
- M. Nejati, S. Samavi and S. Shirani, “Multi-focus image fusion using dictionary-based sparse representation,” Information Fusion, vol. 25, pp. 72–84, 2015. doi
- Z. Wang, A. C. Bovik, H. R. Sheikh and E. P. Simoncelli, “Image quality assessment: From error visibility to structural similarity,” IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600–612, 2004. doi
- X. Zhang, P. Ye and G. Xiao, “VIFB: A visible and infrared image fusion benchmark,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2020. CVF arXiv
- X. Zhang, “Benchmarking and comparing multi-exposure image fusion algorithms,” Information Fusion, vol. 74, pp. 111–131, 2021. doi arXiv
- A. M. Eskicioglu and P. S. Fisher, “Image quality measures and their performance,” IEEE Transactions on Communications, vol. 43, no. 12, pp. 2959–2965, 1995. doi
- G. Qu, D. Zhang and P. Yan, “Information measure for performance of image fusion,” Electronics Letters, vol. 38, no. 7, pp. 313–315, 2002. doi
- M. Hossny, S. Nahavandi and D. Creighton, “Comments on ‘Information measure for performance of image fusion’,” Electronics Letters, vol. 44, no. 18, pp. 1066–1067, 2008. doi
- C. S. Xydeas and V. Petrović, “Objective image fusion performance measure,” Electronics Letters, vol. 36, no. 4, pp. 308–309, 2000. doi
- G. Piella and H. Heijmans, “A new quality metric for image fusion,” in Proc. IEEE International Conference on Image Processing (ICIP), 2003, vol. 3, pp. III-173–176. doi
- Y. Han, Y. Cai, Y. Cao and X. Xu, “A new image fusion performance metric based on visual information fidelity,” Information Fusion, vol. 14, no. 2, pp. 127–135, 2013. doi
- K. Ma, K. Zeng and Z. Wang, “Perceptual quality assessment for multi-exposure image fusion,” IEEE Transactions on Image Processing, vol. 24, no. 11, pp. 3345–3356, 2015. doi
- Z. Liu, E. Blasch, Z. Xue, J. Zhao, R. Laganiere and W. Wu, “Objective assessment of multiresolution image fusion algorithms for context enhancement in night vision: A comparative study,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 34, no. 1, pp. 94–109, 2012. doi
- J. Liu, X. Fan, Z. Huang, G. Wu, R. Liu, W. Zhong and Z. Luo, “Target-aware dual adversarial learning and a multi-scenario multi-modality benchmark to fuse infrared and visible for object detection,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 5802–5811. CVF arXiv M3FD
- J. Liu, Z. Liu, G. Wu, L. Ma, R. Liu, W. Zhong, Z. Luo and X. Fan, “Multi-interactive feature learning and a full-time multi-modality benchmark for image fusion and segmentation,” in Proc. IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 8115–8124. CVF arXiv
- Y. Zhang, Y. Liu, P. Sun, H. Yan, X. Zhao and L. Zhang, “IFCNN: A general image fusion framework based on convolutional neural network,” Information Fusion, vol. 54, pp. 99–118, 2020. doi
- H. Li and X.-J. Wu, “DenseFuse: A fusion approach to infrared and visible images,” IEEE Transactions on Image Processing, vol. 28, no. 5, pp. 2614–2623, 2019. doi arXiv
- K. R. Prabhakar, V. S. Srikar and R. V. Babu, “DeepFuse: A deep unsupervised approach for exposure fusion with extreme exposure image pairs,” in Proc. IEEE International Conference on Computer Vision (ICCV), 2017, pp. 4724–4732. doi CVF
- H. Xu, J. Ma, J. Jiang, X. Guo and H. Ling, “U2Fusion: A unified unsupervised image fusion network,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 1, pp. 502–518, 2022. doi RoadScene
- L. Tang, J. Yuan and J. Ma, “Image fusion in the loop of high-level vision tasks: A semantic-aware real-time infrared and visible image fusion network,” Information Fusion, vol. 82, pp. 28–42, 2022. doi
- J. Ma, L. Tang, F. Fan, J. Huang, X. Mei and Y. Ma, “SwinFusion: Cross-domain long-range learning for general image fusion via Swin Transformer,” IEEE/CAA Journal of Automatica Sinica, vol. 9, no. 7, pp. 1200–1217, 2022. doi
- X. Zhang and Y. Demiris, “Visible and infrared image fusion using deep learning,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 8, pp. 10535–10554, 2023. doi
- Y. Liu, X. Chen, H. Peng and Z. Wang, “Multi-focus image fusion with a deep convolutional neural network,” Information Fusion, vol. 36, pp. 191–207, 2017. doi
- K. Ma, Z. Duanmu, H. Zhu, Y. Fang and Z. Wang, “Deep guided learning for fast multi-exposure image fusion,” IEEE Transactions on Image Processing, vol. 29, pp. 2808–2819, 2020. doi
- Z. Zhao, H. Bai, J. Zhang, Y. Zhang, S. Xu, Z. Lin, R. Timofte and L. Van Gool, “CDDFuse: Correlation-driven dual-branch feature decomposition for multi-modality image fusion,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 5906–5916. CVF arXiv
- J. Ma, W. Yu, P. Liang, C. Li and J. Jiang, “FusionGAN: A generative adversarial network for infrared and visible image fusion,” Information Fusion, vol. 48, pp. 11–26, 2019. doi · public article transcription
- H. Bai, J. Zhang, Z. Zhao, Y. Wu, L. Deng, Y. Cui, T. Feng and S. Xu, “Task-driven image fusion with learnable fusion loss,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025, pp. 7457–7468. CVF arXiv
- Z. Zhao, H. Bai, Y. Zhu, J. Zhang, S. Xu, Y. Zhang, K. Zhang, D. Meng, R. Timofte and L. Van Gool, “DDFM: Denoising diffusion model for multi-modality image fusion,” in Proc. IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 8082–8093. CVF arXiv
- X. Yi, H. Xu, H. Zhang, L. Tang and J. Ma, “Text-IF: Leveraging semantic text guidance for degradation-aware and interactive image fusion,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 27026–27035. CVF arXiv
- D. Wang, J. Liu, X. Fan and R. Liu, “Unsupervised misaligned infrared and visible image fusion via cross-modality image generation and registration,” in Proc. International Joint Conference on Artificial Intelligence (IJCAI), 2022, pp. 3508–3515. doi arXiv
- L. Tang, Y. Deng, Y. Ma, J. Huang and J. Ma, “SuperFusion: A versatile image registration and fusion network with semantic awareness,” IEEE/CAA Journal of Automatica Sinica, vol. 9, no. 12, pp. 2121–2137, 2022. doi
- A. Toet, “The TNO multiband image data collection,” Data in Brief, vol. 15, pp. 249–251, 2017. doi
- L. Tang, J. Yuan, H. Zhang, X. Jiang and J. Ma, “PIAFusion: A progressive infrared and visible image fusion network based on illumination aware,” Information Fusion, vol. 83–84, pp. 79–92, 2022. doi MSRS
- Q. Ha, K. Watanabe, T. Karasawa, Y. Ushiku and T. Harada, “MFNet: Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes,” in Proc. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2017, pp. 5108–5115. doi
- X. Jia, C. Zhu, M. Li, W. Tang and W. Zhou, “LLVIP: A visible-infrared paired dataset for low-light vision,” in Proc. IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, 2021, pp. 3496–3504. CVF arXiv
- J. Cai, S. Gu and L. Zhang, “Learning a deep single image contrast enhancer from multi-exposure images,” IEEE Transactions on Image Processing, vol. 27, no. 4, pp. 2049–2062, 2018. doi
- X. Zhang, “Deep learning-based multi-focus image fusion: A survey and a comparative study,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 9, 2022. doi
- F. Xu, J. Liu, Y. Song, H. Sun and X. Wang, “Multi-exposure image fusion techniques: A comprehensive review,” Remote Sensing, vol. 14, no. 3, art. 771, 2022. doi
- Z. Zhao, H. Bai, J. Zhang, Y. Zhang, K. Zhang, S. Xu, D. Chen, R. Timofte and L. Van Gool, “Equivariant multi-modality image fusion,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 25912–25921. CVF arXiv
- L. Tang, Y. Wang, M. Gong, Z. Li, Y. Deng, X. Yi, C. Li, H. Xu, H. Zhang and J. Ma, “VideoFusion: A spatio-temporal collaborative network for multi-modal video fusion,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026, pp. 19559–19569. CVF
- N. Yokoya, C. Grohnfeldt and J. Chanussot, “Hyperspectral and multispectral data fusion: A comparative review of the recent literature,” IEEE Geoscience and Remote Sensing Magazine, 2017. doi
- H. R. Sheikh and A. C. Bovik, “Image information and visual quality,” IEEE Transactions on Image Processing, 2006. doi
- H. Xu et al., “U2Fusion,” official implementation accompanying the TPAMI paper, 2022. code
- K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” ICLR, 2015. arXiv
- J. Kirkpatrick et al., “Overcoming catastrophic forgetting in neural networks,” Proceedings of the National Academy of Sciences, 2017. doi
- Y. Zhang et al., “IFCNN,” official implementation and errata accompanying the Information Fusion paper, 2020. code
- K. He, X. Zhang, S. Ren and J. Sun, “Deep residual learning for image recognition,” CVPR, 2016. CVF
- J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li and L. Fei-Fei, “ImageNet: A large-scale hierarchical image database,” CVPR, 2009. doi
- N. Silberman, D. Hoiem, P. Kohli and R. Fergus, “Indoor segmentation and support inference from RGBD images,” ECCV, 2012. project
- P. Dhariwal and A. Nichol, “Diffusion models beat GANs on image synthesis,” NeurIPS, 2021. arXiv
- J. Liu et al., “TarDAL,” official implementation accompanying the CVPR paper, 2022. code
- G. Jocher et al., “YOLOv5,” Ultralytics object-detection software, 2020 onward. code
- J. Ma et al., “FusionGAN,” official implementation accompanying the Information Fusion paper, 2019. code
- L. Tang, J. Yuan and J. Ma, “SeAFusion,” official implementation accompanying the Information Fusion paper, 2022. code
- T. Mertens, J. Kautz and F. Van Reeth, “Exposure Fusion,” in Proc. Pacific Graphics, 2007. paper · author project