完整搜尋頁面 →

SeAFusion:讓語意分割影響融合影像

選讀・DLCV 10・影像融合:互補來源與有用輸出 的論文導讀(第 10/11 篇)。引用編號對應本頁底部的參考文獻。

導覽:本頁與本系列: 導覽:本頁與本系列

本頁內容

補充資料 · 6

  1. 散焦擴散效應
  2. 建構與還原拉普拉斯金字塔
  3. 稀疏表示融合
  4. 互資訊作為融合指標
  5. 代表性融合網路的訓練訊號
  6. VIF 與來源參考的融合分數

論文導讀 · 11

  1. 曝光融合(Mertens、Kautz 與 Van Reeth)
  2. VIFB:可見光與紅外線影像融合基準
  3. Liu 等人的紅外線與可見光融合綜述(TPAMI 2025)
  4. DenseFuse:學習重建,在特徵空間選擇融合規則
  5. DDFM:以概似修正擴散先驗
  6. FusionGAN:熱強度與對抗式可見光先驗
  7. IFCNN:支援可變輸入數的監督式融合
  8. U2Fusion:跨融合任務共用一個無監督模型
  9. SwinFusion:來源內與來源間的注意力
  10. SeAFusion:讓語意分割影響融合影像
  11. TarDAL:同時為外觀與物件偵測融合

電腦視覺深度學習

  1. 02 · 深度影像分類
  2. 03 · 物件偵測:從 R-CNN 到 DETR
  3. 09 · 深度影像復原:低階視覺
查看完整課程樹 →

挑戰

融合圖可能銳利且來源分數佳,卻讓分割系統漏掉行人。SeAFusion [33] 保留來源強度、梯度,再以語意分割提供任務訓練訊號。結果仍是一張可檢視、可給其他系統使用的影像,不只是偵測器內部特徵。

本導讀區分期刊問題設定與已開啟官方 PyTorch 實作 [67] 的精確設定;程式是特定版本的可執行證據,不能拿來推填未取得的原始表格。

方法

融合影像送往分割,語意損失再把訓練訊號傳回融合;分割預測與融合監督具有不同角色。
官方專案架構圖 — 融合影像送往分割,語意損失再把訓練訊號傳回融合;分割預測與融合監督具有不同角色。

圖片來源:[33],官方實作圖,版本 cca928453e6969b0c37ea9fbe172ee51aa63c8b8. 原始來源.

雙分支保留模態差異。 可見光 RGB 轉亮度、色度,以亮度與單通道紅外線融合。各分支先用 3×33\times3 轉 16 通道,再經兩個梯度殘差密集區塊轉 32、48。批次 BB、尺寸 H,WH,W 時,各末端為 B×48×H×WB\times48\times H\times W,步幅一與填補保持對齊。

區塊內,密集卷積分支串接中間特徵,再用 1×11\times1 控通道;平行分支用水平、垂直 Sobel 形濾波,加總絕對響應後以另一 1×11\times1 投影。相加後過 LeakyReLU。密集路徑學脈絡,梯度路徑強調小型即時網路可能抹平的邊界。

實作細節:模型內 Sobel 形卷積由濾波器初始化,但在此版本是可學參數;損失內的 Sobel 才明確固定。「梯度引導」不代表所有同名濾波都不更新。

兩個 48 通道串成 96,再經四卷積轉 64、32、16、1,末端轉換 tanh 到 [0,1][0,1]。ConvBnLeakyRelu2d、ConvBnTanh2d 雖建立批次正規化,實際 forward 卻未呼叫。重現應看執行運算,不依類名猜測。最後用原可見光色度還原彩圖。

來源保留基礎目標。 公開程式為

g(X)=∣Kx∗X∣+∣Ky∗X∣,Li=MAE⁡(F,max⁡(Y,I)),Lg=MAE⁡(g(F),max⁡(g(Y),g(I))),Lfusion=Li+10Lg.\begin{aligned} g(X)&=|K_x*X|+|K_y*X|,\\ \mathcal L_i&=\operatorname{MAE}(F,\max(Y,I)),\\ \mathcal L_g&=\operatorname{MAE}(g(F),\max(g(Y),g(I))),\\ \mathcal L_{\rm fusion}&=\mathcal L_i+10\mathcal L_g. \end{aligned}

YY 是可見光亮度,II 是紅外線,FF 是融合亮度,Kx,KyK_x,K_y 是固定 Sobel,∗* 表卷積,MAE 平均絕對殘差;最大值、絕對值逐元素。強度目標保任一來源亮證據,梯度目標保強邊緣,都是人工選定的無監督目標,不是量測理想圖。

分割訊號。 分割模型有主要與輔助預測。類別標籤 SS、主要 logits ZZ、輔助 logits ZaZ_a、困難樣本交叉熵 ℓ\ell 下,融合訓練語意項為

Lsem=ℓ(Z,S)+0.1ℓ(Za,S),Ltotal=Lfusion+rLsem.\mathcal L_{\rm sem}=\ell(Z,S)+0.1\ell(Z_a,S),\qquad \mathcal L_{\rm total}=\mathcal L_{\rm fusion}+r\mathcal L_{\rm sem}.

rr 是該版本的訓練輪係數。困難樣本挖掘強調難分類像素。融合更新時固定分割權重,但對融合輸入仍可微,梯度穿過它傳回融合網路;固定參數不等於關閉計算圖。

外層先產融合圖、訓練或更新分割,再以語意回饋訓練融合。分割訓練的輔助係數是 0.75,融合回饋的輔助係數是 0.1,屬不同階段。首輪只用來源目標,後輪加入任務訊號;不能說成全程相同的同步聯合損失。

雖不需融合真值,語意標籤使任務訓練在分割層面仍屬監督式。部署產圖只需融合網路,不必每幀執行訓練分割器。

結果

官方專案展示 MFNet 融合例,以及紅外線、可見光、融合分割結果,並區分重訓分割器與固定外部分割器,也展示固定偵測器。重訓衡量表示可學到什麼,固定外部模型則衡量對該模型的轉移,問題不同。

先計算損失,再訓練網路。 此處尚未核對原文的數值表。以下 CPU 範例以合成條紋與明亮方塊,檢查公開實作的來源保留目標 [67]。它評估四張固定候選圖,沒有訓練或評比 SeAFusion。

程式碼 · Python · 20 行
import torch
import torch.nn.functional as F

x = torch.arange(32).reshape(1, 1, 1, 32)
y = (0.2 + 0.4 * ((x // 4) % 2)).expand(1, 1, 32, 32).clone()
i = torch.full_like(y, 0.1)
i[:, :, 8:24, 8:24] = 0.9
kx = torch.tensor([[-1., 0., 1.], [-2., 0., 2.], [-1., 0., 1.]])[None, None]
ky = kx.transpose(-1, -2)

def gradient(z):
    return F.conv2d(z, kx, padding=1).abs() + F.conv2d(z, ky, padding=1).abs()

intensity_target = torch.maximum(y, i)
gradient_target = torch.maximum(gradient(y), gradient(i))
for name, fused in [('visible', y), ('infrared', i),
                    ('mean', (y + i) / 2), ('maximum', intensity_target)]:
    li = F.l1_loss(fused, intensity_target)
    lg = F.l1_loss(gradient(fused), gradient_target)
    print(f'{name:8s} intensity={li:.4f} gradient={lg:.4f} total={li + 10 * lg:.4f}')

實際執行輸出:

程式碼 · 4 行
visible  intensity=0.1250 gradient=0.2500 total=2.6250
infrared intensity=0.2250 gradient=0.6867 total=7.0922
mean     intensity=0.1750 gradient=0.5215 total=5.3898
maximum  intensity=0.0000 gradient=0.2625 total=2.6250

最大值圖完全符合強度目標,梯度損失卻仍大於零:空間微分與取最大值不能交換順序。只用可見光與最大值融合甚至得到相同加權總分,保留的來源證據卻不同。此目標不能保證語意用途;仍須查看兩模態,另行評估任務模型。以上數字既非原文 mIoU,也非網路消融結果。

消融實驗

可分別測:保留網路資料而移除語意回饋;盡量匹配解碼器容量而移除梯度分支;保留架構而移除梯度損失。三者各測任務回饋、架構特徵、最佳化目標;一次全移除無法歸因。

首輪與後輪訓練歷程不同,不能直接當乾淨語意消融。需要匹配資料、初始化、訓練預算與分割評估。這些是設計分析,不是虛構原始消融數值,本導讀不填入未取得的表值。

結論

SeAFusion 連接小型雙分支與語意監督。要區分學習式局部特徵、固定來源保留損失、任務網路回饋,才能理解如何沒有融合照片真值,仍同時最佳化可見證據與下游用途。

限制與問題

語意監督可能偏向已標註類別與訓練域,忽略陌生物件;最大強度與梯度也可能保雜訊、熱背景、錯位。單一資料集的分割增益,不證明通用感知品質、偵測效益或跨感測器穩健性。

手機部署須在實際解析度、精度量匯出模型,小架構不保證延遲達標。可比較 TarDAL 的偵測合作訓練;主文指標反例則說明影像與任務評估都要做。

參考文獻

  1. Y. Liu, X. Chen, Z. Wang, Z. J. Wang, R. K. Ward and X. Wang, “Deep learning for pixel-level image fusion: Recent advances and future prospects,” Information Fusion, vol. 42, pp. 158–173, 2018. doi
  2. J. Liu, G. Wu, Z. Liu, D. Wang, Z. Jiang, L. Ma, W. Zhong, X. Fan and R. Liu, “Infrared and visible image fusion: From data compatibility to task adaption,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 47, no. 4, pp. 2349–2369, 2025. doi arXiv
  3. J. Ma, Y. Ma and C. Li, “Infrared and visible image fusion methods and applications: A survey,” Information Fusion, vol. 45, pp. 153–178, 2019. doi
  4. P. E. Debevec and J. Malik, “Recovering high dynamic range radiance maps from photographs,” in Proc. SIGGRAPH, 1997, pp. 369–378. doi
  5. T. Mertens, J. Kautz and F. Van Reeth, “Exposure fusion: A simple and practical alternative to high dynamic range photography,” Computer Graphics Forum, vol. 28, no. 1, pp. 161–171, 2009. doi
  6. S. Xu, X. Wei, C. Zhang, J. Liu and J. Zhang, “MFFW: A new dataset for multi-focus image fusion,” arXiv:2002.04780, 2020. arXiv
  7. X. Zhang, “Multi-focus image fusion: A benchmark,” arXiv:2005.01116, 2020. arXiv
  8. P. J. Burt and E. H. Adelson, “The Laplacian pyramid as a compact image code,” IEEE Transactions on Communications, vol. 31, no. 4, pp. 532–540, 1983. doi
  9. P. J. Burt and R. J. Kolczynski, “Enhanced image capture through fusion,” in Proc. 4th International Conference on Computer Vision (ICCV), 1993, pp. 173–182. doi
  10. P. J. Burt and E. H. Adelson, “A multiresolution spline with application to image mosaics,” ACM Transactions on Graphics, vol. 2, no. 4, pp. 217–236, 1983. doi
  11. H. Li, B. S. Manjunath and S. K. Mitra, “Multisensor image fusion using the wavelet transform,” Graphical Models and Image Processing, vol. 57, no. 3, pp. 235–245, 1995. doi
  12. S. Li, X. Kang and J. Hu, “Image fusion with guided filtering,” IEEE Transactions on Image Processing, vol. 22, no. 7, pp. 2864–2875, 2013. doi
  13. K. He, J. Sun and X. Tang, “Guided image filtering,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, no. 6, pp. 1397–1409, 2013. doi
  14. B. Yang and S. Li, “Multifocus image fusion and restoration with sparse representation,” IEEE Transactions on Instrumentation and Measurement, vol. 59, no. 4, pp. 884–892, 2010. doi
  15. M. Nejati, S. Samavi and S. Shirani, “Multi-focus image fusion using dictionary-based sparse representation,” Information Fusion, vol. 25, pp. 72–84, 2015. doi
  16. Z. Wang, A. C. Bovik, H. R. Sheikh and E. P. Simoncelli, “Image quality assessment: From error visibility to structural similarity,” IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600–612, 2004. doi
  17. X. Zhang, P. Ye and G. Xiao, “VIFB: A visible and infrared image fusion benchmark,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2020. CVF arXiv
  18. X. Zhang, “Benchmarking and comparing multi-exposure image fusion algorithms,” Information Fusion, vol. 74, pp. 111–131, 2021. doi arXiv
  19. A. M. Eskicioglu and P. S. Fisher, “Image quality measures and their performance,” IEEE Transactions on Communications, vol. 43, no. 12, pp. 2959–2965, 1995. doi
  20. G. Qu, D. Zhang and P. Yan, “Information measure for performance of image fusion,” Electronics Letters, vol. 38, no. 7, pp. 313–315, 2002. doi
  21. M. Hossny, S. Nahavandi and D. Creighton, “Comments on ‘Information measure for performance of image fusion’,” Electronics Letters, vol. 44, no. 18, pp. 1066–1067, 2008. doi
  22. C. S. Xydeas and V. Petrović, “Objective image fusion performance measure,” Electronics Letters, vol. 36, no. 4, pp. 308–309, 2000. doi
  23. G. Piella and H. Heijmans, “A new quality metric for image fusion,” in Proc. IEEE International Conference on Image Processing (ICIP), 2003, vol. 3, pp. III-173–176. doi
  24. Y. Han, Y. Cai, Y. Cao and X. Xu, “A new image fusion performance metric based on visual information fidelity,” Information Fusion, vol. 14, no. 2, pp. 127–135, 2013. doi
  25. K. Ma, K. Zeng and Z. Wang, “Perceptual quality assessment for multi-exposure image fusion,” IEEE Transactions on Image Processing, vol. 24, no. 11, pp. 3345–3356, 2015. doi
  26. Z. Liu, E. Blasch, Z. Xue, J. Zhao, R. Laganiere and W. Wu, “Objective assessment of multiresolution image fusion algorithms for context enhancement in night vision: A comparative study,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 34, no. 1, pp. 94–109, 2012. doi
  27. J. Liu, X. Fan, Z. Huang, G. Wu, R. Liu, W. Zhong and Z. Luo, “Target-aware dual adversarial learning and a multi-scenario multi-modality benchmark to fuse infrared and visible for object detection,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 5802–5811. CVF arXiv M3FD
  28. J. Liu, Z. Liu, G. Wu, L. Ma, R. Liu, W. Zhong, Z. Luo and X. Fan, “Multi-interactive feature learning and a full-time multi-modality benchmark for image fusion and segmentation,” in Proc. IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 8115–8124. CVF arXiv
  29. Y. Zhang, Y. Liu, P. Sun, H. Yan, X. Zhao and L. Zhang, “IFCNN: A general image fusion framework based on convolutional neural network,” Information Fusion, vol. 54, pp. 99–118, 2020. doi
  30. H. Li and X.-J. Wu, “DenseFuse: A fusion approach to infrared and visible images,” IEEE Transactions on Image Processing, vol. 28, no. 5, pp. 2614–2623, 2019. doi arXiv
  31. K. R. Prabhakar, V. S. Srikar and R. V. Babu, “DeepFuse: A deep unsupervised approach for exposure fusion with extreme exposure image pairs,” in Proc. IEEE International Conference on Computer Vision (ICCV), 2017, pp. 4724–4732. doi CVF
  32. H. Xu, J. Ma, J. Jiang, X. Guo and H. Ling, “U2Fusion: A unified unsupervised image fusion network,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 1, pp. 502–518, 2022. doi RoadScene
  33. L. Tang, J. Yuan and J. Ma, “Image fusion in the loop of high-level vision tasks: A semantic-aware real-time infrared and visible image fusion network,” Information Fusion, vol. 82, pp. 28–42, 2022. doi
  34. J. Ma, L. Tang, F. Fan, J. Huang, X. Mei and Y. Ma, “SwinFusion: Cross-domain long-range learning for general image fusion via Swin Transformer,” IEEE/CAA Journal of Automatica Sinica, vol. 9, no. 7, pp. 1200–1217, 2022. doi
  35. X. Zhang and Y. Demiris, “Visible and infrared image fusion using deep learning,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 8, pp. 10535–10554, 2023. doi
  36. Y. Liu, X. Chen, H. Peng and Z. Wang, “Multi-focus image fusion with a deep convolutional neural network,” Information Fusion, vol. 36, pp. 191–207, 2017. doi
  37. K. Ma, Z. Duanmu, H. Zhu, Y. Fang and Z. Wang, “Deep guided learning for fast multi-exposure image fusion,” IEEE Transactions on Image Processing, vol. 29, pp. 2808–2819, 2020. doi
  38. Z. Zhao, H. Bai, J. Zhang, Y. Zhang, S. Xu, Z. Lin, R. Timofte and L. Van Gool, “CDDFuse: Correlation-driven dual-branch feature decomposition for multi-modality image fusion,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 5906–5916. CVF arXiv
  39. J. Ma, W. Yu, P. Liang, C. Li and J. Jiang, “FusionGAN: A generative adversarial network for infrared and visible image fusion,” Information Fusion, vol. 48, pp. 11–26, 2019. doi · public article transcription
  40. H. Bai, J. Zhang, Z. Zhao, Y. Wu, L. Deng, Y. Cui, T. Feng and S. Xu, “Task-driven image fusion with learnable fusion loss,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025, pp. 7457–7468. CVF arXiv
  41. Z. Zhao, H. Bai, Y. Zhu, J. Zhang, S. Xu, Y. Zhang, K. Zhang, D. Meng, R. Timofte and L. Van Gool, “DDFM: Denoising diffusion model for multi-modality image fusion,” in Proc. IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 8082–8093. CVF arXiv
  42. X. Yi, H. Xu, H. Zhang, L. Tang and J. Ma, “Text-IF: Leveraging semantic text guidance for degradation-aware and interactive image fusion,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 27026–27035. CVF arXiv
  43. D. Wang, J. Liu, X. Fan and R. Liu, “Unsupervised misaligned infrared and visible image fusion via cross-modality image generation and registration,” in Proc. International Joint Conference on Artificial Intelligence (IJCAI), 2022, pp. 3508–3515. doi arXiv
  44. L. Tang, Y. Deng, Y. Ma, J. Huang and J. Ma, “SuperFusion: A versatile image registration and fusion network with semantic awareness,” IEEE/CAA Journal of Automatica Sinica, vol. 9, no. 12, pp. 2121–2137, 2022. doi
  45. A. Toet, “The TNO multiband image data collection,” Data in Brief, vol. 15, pp. 249–251, 2017. doi
  46. L. Tang, J. Yuan, H. Zhang, X. Jiang and J. Ma, “PIAFusion: A progressive infrared and visible image fusion network based on illumination aware,” Information Fusion, vol. 83–84, pp. 79–92, 2022. doi MSRS
  47. Q. Ha, K. Watanabe, T. Karasawa, Y. Ushiku and T. Harada, “MFNet: Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes,” in Proc. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2017, pp. 5108–5115. doi
  48. X. Jia, C. Zhu, M. Li, W. Tang and W. Zhou, “LLVIP: A visible-infrared paired dataset for low-light vision,” in Proc. IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, 2021, pp. 3496–3504. CVF arXiv
  49. J. Cai, S. Gu and L. Zhang, “Learning a deep single image contrast enhancer from multi-exposure images,” IEEE Transactions on Image Processing, vol. 27, no. 4, pp. 2049–2062, 2018. doi
  50. X. Zhang, “Deep learning-based multi-focus image fusion: A survey and a comparative study,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 9, 2022. doi
  51. F. Xu, J. Liu, Y. Song, H. Sun and X. Wang, “Multi-exposure image fusion techniques: A comprehensive review,” Remote Sensing, vol. 14, no. 3, art. 771, 2022. doi
  52. Z. Zhao, H. Bai, J. Zhang, Y. Zhang, K. Zhang, S. Xu, D. Chen, R. Timofte and L. Van Gool, “Equivariant multi-modality image fusion,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 25912–25921. CVF arXiv
  53. L. Tang, Y. Wang, M. Gong, Z. Li, Y. Deng, X. Yi, C. Li, H. Xu, H. Zhang and J. Ma, “VideoFusion: A spatio-temporal collaborative network for multi-modal video fusion,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026, pp. 19559–19569. CVF
  54. N. Yokoya, C. Grohnfeldt and J. Chanussot, “Hyperspectral and multispectral data fusion: A comparative review of the recent literature,” IEEE Geoscience and Remote Sensing Magazine, 2017. doi
  55. H. R. Sheikh and A. C. Bovik, “Image information and visual quality,” IEEE Transactions on Image Processing, 2006. doi
  56. H. Xu et al., “U2Fusion,” official implementation accompanying the TPAMI paper, 2022. code
  57. K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” ICLR, 2015. arXiv
  58. J. Kirkpatrick et al., “Overcoming catastrophic forgetting in neural networks,” Proceedings of the National Academy of Sciences, 2017. doi
  59. Y. Zhang et al., “IFCNN,” official implementation and errata accompanying the Information Fusion paper, 2020. code
  60. K. He, X. Zhang, S. Ren and J. Sun, “Deep residual learning for image recognition,” CVPR, 2016. CVF
  61. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li and L. Fei-Fei, “ImageNet: A large-scale hierarchical image database,” CVPR, 2009. doi
  62. N. Silberman, D. Hoiem, P. Kohli and R. Fergus, “Indoor segmentation and support inference from RGBD images,” ECCV, 2012. project
  63. P. Dhariwal and A. Nichol, “Diffusion models beat GANs on image synthesis,” NeurIPS, 2021. arXiv
  64. J. Liu et al., “TarDAL,” official implementation accompanying the CVPR paper, 2022. code
  65. G. Jocher et al., “YOLOv5,” Ultralytics object-detection software, 2020 onward. code
  66. J. Ma et al., “FusionGAN,” official implementation accompanying the Information Fusion paper, 2019. code
  67. L. Tang, J. Yuan and J. Ma, “SeAFusion,” official implementation accompanying the Information Fusion paper, 2022. code
  68. T. Mertens, J. Kautz and F. Van Reeth, “Exposure Fusion,” in Proc. Pacific Graphics, 2007. paper · author project