完整搜尋頁面 →

SwinFusion:來源內與來源間的注意力

選讀・DLCV 10・影像融合:互補來源與有用輸出 的論文導讀(第 9/11 篇)。引用編號對應本頁底部的參考文獻。

導覽:本頁與本系列: 導覽:本頁與本系列

本頁內容

補充資料 · 6

  1. 散焦擴散效應
  2. 建構與還原拉普拉斯金字塔
  3. 稀疏表示融合
  4. 互資訊作為融合指標
  5. 代表性融合網路的訓練訊號
  6. VIF 與來源參考的融合分數

論文導讀 · 11

  1. 曝光融合(Mertens、Kautz 與 Van Reeth)
  2. VIFB:可見光與紅外線影像融合基準
  3. Liu 等人的紅外線與可見光融合綜述(TPAMI 2025)
  4. DenseFuse:學習重建,在特徵空間選擇融合規則
  5. DDFM:以概似修正擴散先驗
  6. FusionGAN:熱強度與對抗式可見光先驗
  7. IFCNN:支援可變輸入數的監督式融合
  8. U2Fusion:跨融合任務共用一個無監督模型
  9. SwinFusion:來源內與來源間的注意力
  10. SeAFusion:讓語意分割影響融合影像
  11. TarDAL:同時為外觀與物件偵測融合

電腦視覺深度學習

  1. 02 · 深度影像分類
  2. 03 · 物件偵測:從 R-CNN 到 DETR
  3. 09 · 深度影像復原:低階視覺
查看完整課程樹 →

挑戰

有用結構不一定落在小卷積鄰域:暗行人需脈絡才能與熱背景區分,曝光邊界也可能需跨物體一致決策。來源外觀又不同,來源內注意力與跨來源注意力各有角色。SwinFusion [34] 結合兩者,處理紅外線—可見光、可見光—近紅外線、醫學、多焦點與多曝光。

此處「通用」指可重用架構與目標;各任務分別訓練,不證明同一固定檢查點通吃所有模態,有別於 U2Fusion 的持續學習。

方法

自注意力先處理各來源內部,再以交叉注意力交換來源資訊,最後重建;請分開追蹤查詢、鍵、值路徑。
原論文圖 2(官方專案提供) — 自注意力先處理各來源內部,再以交叉注意力交換來源資訊,最後重建;請分開追蹤查詢、鍵、值路徑。

圖片來源:[34],官方實作提供的原論文圖 2,版本 82bf7381338f90a1d056194e5b1f16cf14f01d8f. 原始來源.

淺層保留局部證據。 各來源先過兩個步幅一 3×33\times3、LeakyReLU 卷積。對齊灰階批次 B×1×H×WB\times1\times H\times W 變為 B×C×H×WB\times C\times H\times W;BB 是批次、CC 是特徵寬度、H,WH,W 是空間尺寸。注意力使用局部特徵,而非只比較原始亮度。

深層擴大脈絡。 四層 Swin Transformer 做來源內擷取。預設視窗邊長八、六個頭;寬 MM 的視窗有 M2M^2 個 token。單頭為

Q=XWQ,K=XWK,V=XWV,Attn⁡(Q,K,V)=softmax⁡ ⁣(QK⊤dk+Br)V.Q=XW_Q,\quad K=XW_K,\quad V=XW_V,\qquad \operatorname{Attn}(Q,K,V)=\operatorname{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}+B_r\right)V.

X∈RM2×CX\in\mathbb R^{M^2\times C} 是視窗特徵,WQ,WK,WVW_Q,W_K,W_V 是投影,Q,K,VQ,K,V 是查詢、鍵、值,dkd_k 是鍵寬,BrB_r 是相對位置偏差。Softmax 對各查詢的候選鍵運算,多頭結果重組;前饋層轉通道,正規化與殘差穩定更新。

一般視窗與位移 ⌊M/2⌋\lfloor M/2\rfloor 的視窗交替,使資訊跨原邊界。這是多層逐步擴展脈絡,不是一次全域兩兩注意力。矩陣大小依視窗面積而非整圖面積,但仍有計算成本與邊界處理。

跨域融合 ACFM。 先以域內自注意力修整各來源,再以域間注意力交換。從來源 2 更新來源 1 時,查詢來自 1,鍵與值來自 2:

Z1←2=Attn⁡(Q1,K2,V2).Z_{1\leftarrow2}=\operatorname{Attn}(Q_1,K_2,V_2).

下標指出來源,Z1←2Z_{1\leftarrow2} 是匯入更新;反方向也計算。殘差保留原來源,跨注意力是更新而非強制替代。使用兩個 ACFM 階段,再串接並卷積聚合;特徵匹配不能取代幾何配準。

重建。 融合深特徵再過四層 Swin,三個步幅一 3×33\times3 卷積縮通道重建,前兩個用 LeakyReLU。因此融合與重建都利用較大脈絡;沒有判別器、擴散取樣器或預訓練 VGG 損失。

三個保留損失。 對融合 FF 與對齊來源 I1,I2I_1,I_2,結構項為

Ls=12[1−SSIM⁡(F,I1)]+12[1−SSIM⁡(F,I2)].\mathcal L_s=\tfrac12[1-\operatorname{SSIM}(F,I_1)]+\tfrac12[1-\operatorname{SSIM}(F,I_2)].

SSIM 如前定義,兩來源係數相同表示結構同等重要;紋理與強度項為

Lt=1HW∥∣∇F∣−max⁡(∣∇I1∣,∣∇I2∣)∥1,Li=1HW∥F−M(I1,I2)∥1,L=10Ls+20Lt+20Li.\begin{aligned} \mathcal L_t&=\frac1{HW}\left\|\lvert\nabla F\rvert-\max(\lvert\nabla I_1\rvert,\lvert\nabla I_2\rvert)\right\|_1,\\ \mathcal L_i&=\frac1{HW}\|F-\mathcal M(I_1,I_2)\|_1,\\ \mathcal L&=10\mathcal L_s+20\mathcal L_t+20\mathcal L_i. \end{aligned}

∇\nabla 是 Sobel,絕對值與最大值逐元素,∥⋅∥1\|\cdot\|_1 加總絕對殘差。M\mathcal M 在紅外線—可見光、醫學、多焦點取最大,在可見光—近紅外線、多曝光取平均。最大強度可保亮目標,最大梯度可保任一來源強邊緣,但也會偏雜訊;都不是物理真值。

每任務採 128×128128\times128 裁切、批次 16、10,000 步、Adam、初始學習率 2×10−42\times10^{-4}。這是原實驗設定,不是任意感測資料的必用配方。

結果

表 I 紅外線—可見光:SwinFusion 的 FMI 0.9308、QAB/FQ^{AB/F} 0.6428、SSIM 0.9676,U2Fusion 為 0.9135、0.4218、0.9387。同表 PSNR 卻是 SwinFusion 64.4063、DenseFuse 66.6532,並非全面勝出。來源誤差也可能獎勵只保留一來源、未整合互補證據。

CT–MRI 的 SSIM 為 0.9497,低於 IFCNN 0.9564;QAB/FQ^{AB/F} 卻為 0.6529,高於 IFCNN 0.6233。應連同輸出看各模態保留準則的取捨,不能當作臨床效益或完整診斷結構保證。

表 VI 紅外線—可見光的參數 0.9737 M、時間 1.4668 秒,IFCNN 為 0.0836 M/0.0130 秒。使用視窗不代表最快,時間也不能移植到其他尺寸或裝置。

消融實驗

表 V 分開移除深層擷取 DE、整個 ACFM、域間互動與各損失。紅外線—可見光 QAB/FQ^{AB/F} 從完整 0.6428,移除 DE 降為 0.3355,移除 ACFM 為 0.4592,只移除域間為 0.5612。單靠來源內脈絡無法重現交換資訊的效益。

移除 SSIM、紋理、強度後,紅外線—可見光 QAB/FQ^{AB/F} 分別 0.2374、0.3809、0.4742。多曝光完整 SSIM 0.9885,移除紋理與強度為 0.8587、0.8231,顯示各項約束不同失敗情況。

移除紋理使紅外線—可見光 PSNR 64.4063→67.0619,卻降低邊緣傳遞品質。表 V 完整 SSIM 印為 0.96976,表 I 卻為 0.9676;此處保留主要比較值,不擅自調和不一致。

結論

SwinFusion 結合來源內、來源間視窗注意力、重建與任務調整的保留損失。實驗支持架構與目標互補,也明示速度及指標取捨;它是多任務框架,不是單一通用權重或品質分數的證明。

限制與問題

視窗注意力不會解決錯位、模糊、缺測或對抗污染。保留損失無法判斷兩來源都退化時誰正確;注意力可能傳播錯誤對應,強度目標也可能保留不需要的熱背景或曝光偽影。

任務聚合與訓練資料也是方法的一部分。若轉用高光譜重建或診斷流程,須另建量測與評估論證。影像先驗可比較 DDFM,明確下游目標則見 TarDAL。

參考文獻

  1. Y. Liu, X. Chen, Z. Wang, Z. J. Wang, R. K. Ward and X. Wang, “Deep learning for pixel-level image fusion: Recent advances and future prospects,” Information Fusion, vol. 42, pp. 158–173, 2018. doi
  2. J. Liu, G. Wu, Z. Liu, D. Wang, Z. Jiang, L. Ma, W. Zhong, X. Fan and R. Liu, “Infrared and visible image fusion: From data compatibility to task adaption,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 47, no. 4, pp. 2349–2369, 2025. doi arXiv
  3. J. Ma, Y. Ma and C. Li, “Infrared and visible image fusion methods and applications: A survey,” Information Fusion, vol. 45, pp. 153–178, 2019. doi
  4. P. E. Debevec and J. Malik, “Recovering high dynamic range radiance maps from photographs,” in Proc. SIGGRAPH, 1997, pp. 369–378. doi
  5. T. Mertens, J. Kautz and F. Van Reeth, “Exposure fusion: A simple and practical alternative to high dynamic range photography,” Computer Graphics Forum, vol. 28, no. 1, pp. 161–171, 2009. doi
  6. S. Xu, X. Wei, C. Zhang, J. Liu and J. Zhang, “MFFW: A new dataset for multi-focus image fusion,” arXiv:2002.04780, 2020. arXiv
  7. X. Zhang, “Multi-focus image fusion: A benchmark,” arXiv:2005.01116, 2020. arXiv
  8. P. J. Burt and E. H. Adelson, “The Laplacian pyramid as a compact image code,” IEEE Transactions on Communications, vol. 31, no. 4, pp. 532–540, 1983. doi
  9. P. J. Burt and R. J. Kolczynski, “Enhanced image capture through fusion,” in Proc. 4th International Conference on Computer Vision (ICCV), 1993, pp. 173–182. doi
  10. P. J. Burt and E. H. Adelson, “A multiresolution spline with application to image mosaics,” ACM Transactions on Graphics, vol. 2, no. 4, pp. 217–236, 1983. doi
  11. H. Li, B. S. Manjunath and S. K. Mitra, “Multisensor image fusion using the wavelet transform,” Graphical Models and Image Processing, vol. 57, no. 3, pp. 235–245, 1995. doi
  12. S. Li, X. Kang and J. Hu, “Image fusion with guided filtering,” IEEE Transactions on Image Processing, vol. 22, no. 7, pp. 2864–2875, 2013. doi
  13. K. He, J. Sun and X. Tang, “Guided image filtering,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, no. 6, pp. 1397–1409, 2013. doi
  14. B. Yang and S. Li, “Multifocus image fusion and restoration with sparse representation,” IEEE Transactions on Instrumentation and Measurement, vol. 59, no. 4, pp. 884–892, 2010. doi
  15. M. Nejati, S. Samavi and S. Shirani, “Multi-focus image fusion using dictionary-based sparse representation,” Information Fusion, vol. 25, pp. 72–84, 2015. doi
  16. Z. Wang, A. C. Bovik, H. R. Sheikh and E. P. Simoncelli, “Image quality assessment: From error visibility to structural similarity,” IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600–612, 2004. doi
  17. X. Zhang, P. Ye and G. Xiao, “VIFB: A visible and infrared image fusion benchmark,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2020. CVF arXiv
  18. X. Zhang, “Benchmarking and comparing multi-exposure image fusion algorithms,” Information Fusion, vol. 74, pp. 111–131, 2021. doi arXiv
  19. A. M. Eskicioglu and P. S. Fisher, “Image quality measures and their performance,” IEEE Transactions on Communications, vol. 43, no. 12, pp. 2959–2965, 1995. doi
  20. G. Qu, D. Zhang and P. Yan, “Information measure for performance of image fusion,” Electronics Letters, vol. 38, no. 7, pp. 313–315, 2002. doi
  21. M. Hossny, S. Nahavandi and D. Creighton, “Comments on ‘Information measure for performance of image fusion’,” Electronics Letters, vol. 44, no. 18, pp. 1066–1067, 2008. doi
  22. C. S. Xydeas and V. Petrović, “Objective image fusion performance measure,” Electronics Letters, vol. 36, no. 4, pp. 308–309, 2000. doi
  23. G. Piella and H. Heijmans, “A new quality metric for image fusion,” in Proc. IEEE International Conference on Image Processing (ICIP), 2003, vol. 3, pp. III-173–176. doi
  24. Y. Han, Y. Cai, Y. Cao and X. Xu, “A new image fusion performance metric based on visual information fidelity,” Information Fusion, vol. 14, no. 2, pp. 127–135, 2013. doi
  25. K. Ma, K. Zeng and Z. Wang, “Perceptual quality assessment for multi-exposure image fusion,” IEEE Transactions on Image Processing, vol. 24, no. 11, pp. 3345–3356, 2015. doi
  26. Z. Liu, E. Blasch, Z. Xue, J. Zhao, R. Laganiere and W. Wu, “Objective assessment of multiresolution image fusion algorithms for context enhancement in night vision: A comparative study,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 34, no. 1, pp. 94–109, 2012. doi
  27. J. Liu, X. Fan, Z. Huang, G. Wu, R. Liu, W. Zhong and Z. Luo, “Target-aware dual adversarial learning and a multi-scenario multi-modality benchmark to fuse infrared and visible for object detection,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 5802–5811. CVF arXiv M3FD
  28. J. Liu, Z. Liu, G. Wu, L. Ma, R. Liu, W. Zhong, Z. Luo and X. Fan, “Multi-interactive feature learning and a full-time multi-modality benchmark for image fusion and segmentation,” in Proc. IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 8115–8124. CVF arXiv
  29. Y. Zhang, Y. Liu, P. Sun, H. Yan, X. Zhao and L. Zhang, “IFCNN: A general image fusion framework based on convolutional neural network,” Information Fusion, vol. 54, pp. 99–118, 2020. doi
  30. H. Li and X.-J. Wu, “DenseFuse: A fusion approach to infrared and visible images,” IEEE Transactions on Image Processing, vol. 28, no. 5, pp. 2614–2623, 2019. doi arXiv
  31. K. R. Prabhakar, V. S. Srikar and R. V. Babu, “DeepFuse: A deep unsupervised approach for exposure fusion with extreme exposure image pairs,” in Proc. IEEE International Conference on Computer Vision (ICCV), 2017, pp. 4724–4732. doi CVF
  32. H. Xu, J. Ma, J. Jiang, X. Guo and H. Ling, “U2Fusion: A unified unsupervised image fusion network,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 1, pp. 502–518, 2022. doi RoadScene
  33. L. Tang, J. Yuan and J. Ma, “Image fusion in the loop of high-level vision tasks: A semantic-aware real-time infrared and visible image fusion network,” Information Fusion, vol. 82, pp. 28–42, 2022. doi
  34. J. Ma, L. Tang, F. Fan, J. Huang, X. Mei and Y. Ma, “SwinFusion: Cross-domain long-range learning for general image fusion via Swin Transformer,” IEEE/CAA Journal of Automatica Sinica, vol. 9, no. 7, pp. 1200–1217, 2022. doi
  35. X. Zhang and Y. Demiris, “Visible and infrared image fusion using deep learning,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 8, pp. 10535–10554, 2023. doi
  36. Y. Liu, X. Chen, H. Peng and Z. Wang, “Multi-focus image fusion with a deep convolutional neural network,” Information Fusion, vol. 36, pp. 191–207, 2017. doi
  37. K. Ma, Z. Duanmu, H. Zhu, Y. Fang and Z. Wang, “Deep guided learning for fast multi-exposure image fusion,” IEEE Transactions on Image Processing, vol. 29, pp. 2808–2819, 2020. doi
  38. Z. Zhao, H. Bai, J. Zhang, Y. Zhang, S. Xu, Z. Lin, R. Timofte and L. Van Gool, “CDDFuse: Correlation-driven dual-branch feature decomposition for multi-modality image fusion,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 5906–5916. CVF arXiv
  39. J. Ma, W. Yu, P. Liang, C. Li and J. Jiang, “FusionGAN: A generative adversarial network for infrared and visible image fusion,” Information Fusion, vol. 48, pp. 11–26, 2019. doi · public article transcription
  40. H. Bai, J. Zhang, Z. Zhao, Y. Wu, L. Deng, Y. Cui, T. Feng and S. Xu, “Task-driven image fusion with learnable fusion loss,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025, pp. 7457–7468. CVF arXiv
  41. Z. Zhao, H. Bai, Y. Zhu, J. Zhang, S. Xu, Y. Zhang, K. Zhang, D. Meng, R. Timofte and L. Van Gool, “DDFM: Denoising diffusion model for multi-modality image fusion,” in Proc. IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 8082–8093. CVF arXiv
  42. X. Yi, H. Xu, H. Zhang, L. Tang and J. Ma, “Text-IF: Leveraging semantic text guidance for degradation-aware and interactive image fusion,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 27026–27035. CVF arXiv
  43. D. Wang, J. Liu, X. Fan and R. Liu, “Unsupervised misaligned infrared and visible image fusion via cross-modality image generation and registration,” in Proc. International Joint Conference on Artificial Intelligence (IJCAI), 2022, pp. 3508–3515. doi arXiv
  44. L. Tang, Y. Deng, Y. Ma, J. Huang and J. Ma, “SuperFusion: A versatile image registration and fusion network with semantic awareness,” IEEE/CAA Journal of Automatica Sinica, vol. 9, no. 12, pp. 2121–2137, 2022. doi
  45. A. Toet, “The TNO multiband image data collection,” Data in Brief, vol. 15, pp. 249–251, 2017. doi
  46. L. Tang, J. Yuan, H. Zhang, X. Jiang and J. Ma, “PIAFusion: A progressive infrared and visible image fusion network based on illumination aware,” Information Fusion, vol. 83–84, pp. 79–92, 2022. doi MSRS
  47. Q. Ha, K. Watanabe, T. Karasawa, Y. Ushiku and T. Harada, “MFNet: Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes,” in Proc. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2017, pp. 5108–5115. doi
  48. X. Jia, C. Zhu, M. Li, W. Tang and W. Zhou, “LLVIP: A visible-infrared paired dataset for low-light vision,” in Proc. IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, 2021, pp. 3496–3504. CVF arXiv
  49. J. Cai, S. Gu and L. Zhang, “Learning a deep single image contrast enhancer from multi-exposure images,” IEEE Transactions on Image Processing, vol. 27, no. 4, pp. 2049–2062, 2018. doi
  50. X. Zhang, “Deep learning-based multi-focus image fusion: A survey and a comparative study,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 9, 2022. doi
  51. F. Xu, J. Liu, Y. Song, H. Sun and X. Wang, “Multi-exposure image fusion techniques: A comprehensive review,” Remote Sensing, vol. 14, no. 3, art. 771, 2022. doi
  52. Z. Zhao, H. Bai, J. Zhang, Y. Zhang, K. Zhang, S. Xu, D. Chen, R. Timofte and L. Van Gool, “Equivariant multi-modality image fusion,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 25912–25921. CVF arXiv
  53. L. Tang, Y. Wang, M. Gong, Z. Li, Y. Deng, X. Yi, C. Li, H. Xu, H. Zhang and J. Ma, “VideoFusion: A spatio-temporal collaborative network for multi-modal video fusion,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026, pp. 19559–19569. CVF
  54. N. Yokoya, C. Grohnfeldt and J. Chanussot, “Hyperspectral and multispectral data fusion: A comparative review of the recent literature,” IEEE Geoscience and Remote Sensing Magazine, 2017. doi
  55. H. R. Sheikh and A. C. Bovik, “Image information and visual quality,” IEEE Transactions on Image Processing, 2006. doi
  56. H. Xu et al., “U2Fusion,” official implementation accompanying the TPAMI paper, 2022. code
  57. K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” ICLR, 2015. arXiv
  58. J. Kirkpatrick et al., “Overcoming catastrophic forgetting in neural networks,” Proceedings of the National Academy of Sciences, 2017. doi
  59. Y. Zhang et al., “IFCNN,” official implementation and errata accompanying the Information Fusion paper, 2020. code
  60. K. He, X. Zhang, S. Ren and J. Sun, “Deep residual learning for image recognition,” CVPR, 2016. CVF
  61. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li and L. Fei-Fei, “ImageNet: A large-scale hierarchical image database,” CVPR, 2009. doi
  62. N. Silberman, D. Hoiem, P. Kohli and R. Fergus, “Indoor segmentation and support inference from RGBD images,” ECCV, 2012. project
  63. P. Dhariwal and A. Nichol, “Diffusion models beat GANs on image synthesis,” NeurIPS, 2021. arXiv
  64. J. Liu et al., “TarDAL,” official implementation accompanying the CVPR paper, 2022. code
  65. G. Jocher et al., “YOLOv5,” Ultralytics object-detection software, 2020 onward. code
  66. J. Ma et al., “FusionGAN,” official implementation accompanying the Information Fusion paper, 2019. code
  67. L. Tang, J. Yuan and J. Ma, “SeAFusion,” official implementation accompanying the Information Fusion paper, 2022. code
  68. T. Mertens, J. Kautz and F. Van Reeth, “Exposure Fusion,” in Proc. Pacific Graphics, 2007. paper · author project