TarDAL:同時為外觀與物件偵測融合
選讀・DLCV 10・影像融合:互補來源與有用輸出 的論文導讀(第 11/11 篇)。引用編號對應本頁底部的參考文獻。
導覽:本頁與本系列: 導覽:本頁與本系列
Liu 等人的 CVPR 2022 論文把偵測納入融合目標 [27]。生成器保共同結構,兩判別器專注熱目標、可見光細節,合作訓練讓偵測梯度影響融合圖。
挑戰
融合可能熵高、外觀好,卻更難偵測;只最佳化偵測又可能不適合人看。TarDAL 同時追求視覺可用與偵測表現。
紅外線可突出暖前景,可見光帶背景紋理。把所有像素一視同仁的單判別器不易表達這些偏好,因此分開內容與表示的對抗比較。
方法

圖片來源: [27], Published paper PDF, PDF 第 3 頁. 原始 PDF.
融合與偵測。 令 為紅外線、 為可見光、 為融合、 為偵測器,實際聯合目標形式為
是框與類別標註, 分別是融合與偵測參數, 是偵測損失, 是融合正規化, 控取捨。這是雙層動機的可訓練形式,不是每步完整解巢狀最佳化。
偵測器採 YOLOv5 [65],梯度經鏈式法則穿過偵測器與生成器到融合參數。融合正規化避免輸出變成任意偵測專用編碼;即使沒有融合真值,合作訓練仍使用偵測標籤。
生成器。 五層密集區塊擷取,三卷積合併;來源共用空間網格,輸出維持尺寸。密集連接重用中間資訊,重建路徑合成影像;推論不需判別器。
目標遮罩。 預訓練顯著性網路由紅外線估熱前景, 強調目標,補集強調背景。這是重要性代理,不是完美標註;遮罩錯誤會改變接受對抗監督的證據。
雙判別器。 比較 與 ,保熱目標強度; 比較可見光與融合的梯度表示,可限制在背景。 是逐元素乘,空間梯度提供邊緣幅度。兩者架構模式相同,但參數、輸入分開。
對抗訓練使用真假分數差與梯度範數正規化。以最大化 critic 慣例,可示意為 ,原文 [27]。生成器則追求高目標與細節分數,轉成最小化程式時須保持符號一致。
來源重建。 SSIM 約束共同結構,顯著性像素保真平衡來源。原文為 , 加權各項 [27]。
重現要注意:PDF 式 5 的顯著性分母印作 ,一般不構成有效非負正規化。目前官方程式其中一路用 、,加權各來源結構與 L1 [64]; 是顯著性估計,非負性依其尺度。其他配置另有來源損失選項。須匹配版本與設定,不能把排版式、現行程式、原發表實驗默認相同。
合作更新。 交替訓練 critic 與融合/偵測。偵測器學融合圖,生成器同時收融合與偵測監督,不同於獨立訓完融合再固定輸出訓偵測。部署可只看融合圖或再送偵測器,報時間須說明哪條路徑。
結果
M3FD 有 4,200 組對齊紅外線—可見光偵測配對,涵蓋白天、陰天、夜間與困難條件,另有融合比較集合 [27]。採集與配準是資料貢獻,模型不會自動對齊任意相機。
表 2 在指定 YOLOv5 設定,Multispectral 的 mAP@0.5 0.613,M3FD 0.807;M3FD 可見光、紅外線單模態為 0.756、0.781 [27]。這依賴偵測器與資料流程,不是獨立於偵測器的融合品質分數。
同表融合模型為 0.296 百萬參數、14.88 GFLOPs、0.041 秒 [27]。須保留量測邊界、解析度與實作,不能當未定義的完整相機到偵測時間。
消融實驗
表 3:無 critic 的 M3FD mAP@0.5 0.722,僅細節 0.719,僅目標 0.781,雙 critic 0.807 [27]。細節 critic 幫助紋理,不表示單獨加入就必然提高 AP。
同表 TNO 無 critic 的 EN 7.223/SD 53.107,高於完整模型 7.177/51.352,但原文展示前者更吵 [27]。熵與離散程度較高不足以證明融合更好。
表 4 的 M3FD 直接、任務導向、合作訓練依序 0.807、0.809、0.811;Multispectral 為 0.613、0.615、0.624 mAP@0.5 [27]。只顧任務可改善偵測卻損外觀,合作追求折衷。圖 10 另以視覺檢查顯著性與遮罩,未完整數值隔離各項。
結論
TarDAL 以具體下游任務定義用途,同時保留視覺限制。雙 critic 表達不同模態貢獻,合作訓練讓生成器適應偵測,而非只追一般影像統計。
限制與問題
融合依賴標籤、顯著性、特定偵測器與資料分布,該偵測器 AP 較好不保證跨架構或任務。原表與現行程式也需仔細匹配版本。
顯示融合圖不是熱量測。學習式強度變化可能幫辨識卻改變定量意義;外觀偽影、偵測錯誤、配準失敗與效率須分開評估,一個勝出指標無法全部認證。
參考文獻
- Y. Liu, X. Chen, Z. Wang, Z. J. Wang, R. K. Ward and X. Wang, “Deep learning for pixel-level image fusion: Recent advances and future prospects,” Information Fusion, vol. 42, pp. 158–173, 2018. doi
- J. Liu, G. Wu, Z. Liu, D. Wang, Z. Jiang, L. Ma, W. Zhong, X. Fan and R. Liu, “Infrared and visible image fusion: From data compatibility to task adaption,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 47, no. 4, pp. 2349–2369, 2025. doi arXiv
- J. Ma, Y. Ma and C. Li, “Infrared and visible image fusion methods and applications: A survey,” Information Fusion, vol. 45, pp. 153–178, 2019. doi
- P. E. Debevec and J. Malik, “Recovering high dynamic range radiance maps from photographs,” in Proc. SIGGRAPH, 1997, pp. 369–378. doi
- T. Mertens, J. Kautz and F. Van Reeth, “Exposure fusion: A simple and practical alternative to high dynamic range photography,” Computer Graphics Forum, vol. 28, no. 1, pp. 161–171, 2009. doi
- S. Xu, X. Wei, C. Zhang, J. Liu and J. Zhang, “MFFW: A new dataset for multi-focus image fusion,” arXiv:2002.04780, 2020. arXiv
- X. Zhang, “Multi-focus image fusion: A benchmark,” arXiv:2005.01116, 2020. arXiv
- P. J. Burt and E. H. Adelson, “The Laplacian pyramid as a compact image code,” IEEE Transactions on Communications, vol. 31, no. 4, pp. 532–540, 1983. doi
- P. J. Burt and R. J. Kolczynski, “Enhanced image capture through fusion,” in Proc. 4th International Conference on Computer Vision (ICCV), 1993, pp. 173–182. doi
- P. J. Burt and E. H. Adelson, “A multiresolution spline with application to image mosaics,” ACM Transactions on Graphics, vol. 2, no. 4, pp. 217–236, 1983. doi
- H. Li, B. S. Manjunath and S. K. Mitra, “Multisensor image fusion using the wavelet transform,” Graphical Models and Image Processing, vol. 57, no. 3, pp. 235–245, 1995. doi
- S. Li, X. Kang and J. Hu, “Image fusion with guided filtering,” IEEE Transactions on Image Processing, vol. 22, no. 7, pp. 2864–2875, 2013. doi
- K. He, J. Sun and X. Tang, “Guided image filtering,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, no. 6, pp. 1397–1409, 2013. doi
- B. Yang and S. Li, “Multifocus image fusion and restoration with sparse representation,” IEEE Transactions on Instrumentation and Measurement, vol. 59, no. 4, pp. 884–892, 2010. doi
- M. Nejati, S. Samavi and S. Shirani, “Multi-focus image fusion using dictionary-based sparse representation,” Information Fusion, vol. 25, pp. 72–84, 2015. doi
- Z. Wang, A. C. Bovik, H. R. Sheikh and E. P. Simoncelli, “Image quality assessment: From error visibility to structural similarity,” IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600–612, 2004. doi
- X. Zhang, P. Ye and G. Xiao, “VIFB: A visible and infrared image fusion benchmark,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2020. CVF arXiv
- X. Zhang, “Benchmarking and comparing multi-exposure image fusion algorithms,” Information Fusion, vol. 74, pp. 111–131, 2021. doi arXiv
- A. M. Eskicioglu and P. S. Fisher, “Image quality measures and their performance,” IEEE Transactions on Communications, vol. 43, no. 12, pp. 2959–2965, 1995. doi
- G. Qu, D. Zhang and P. Yan, “Information measure for performance of image fusion,” Electronics Letters, vol. 38, no. 7, pp. 313–315, 2002. doi
- M. Hossny, S. Nahavandi and D. Creighton, “Comments on ‘Information measure for performance of image fusion’,” Electronics Letters, vol. 44, no. 18, pp. 1066–1067, 2008. doi
- C. S. Xydeas and V. Petrović, “Objective image fusion performance measure,” Electronics Letters, vol. 36, no. 4, pp. 308–309, 2000. doi
- G. Piella and H. Heijmans, “A new quality metric for image fusion,” in Proc. IEEE International Conference on Image Processing (ICIP), 2003, vol. 3, pp. III-173–176. doi
- Y. Han, Y. Cai, Y. Cao and X. Xu, “A new image fusion performance metric based on visual information fidelity,” Information Fusion, vol. 14, no. 2, pp. 127–135, 2013. doi
- K. Ma, K. Zeng and Z. Wang, “Perceptual quality assessment for multi-exposure image fusion,” IEEE Transactions on Image Processing, vol. 24, no. 11, pp. 3345–3356, 2015. doi
- Z. Liu, E. Blasch, Z. Xue, J. Zhao, R. Laganiere and W. Wu, “Objective assessment of multiresolution image fusion algorithms for context enhancement in night vision: A comparative study,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 34, no. 1, pp. 94–109, 2012. doi
- J. Liu, X. Fan, Z. Huang, G. Wu, R. Liu, W. Zhong and Z. Luo, “Target-aware dual adversarial learning and a multi-scenario multi-modality benchmark to fuse infrared and visible for object detection,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 5802–5811. CVF arXiv M3FD
- J. Liu, Z. Liu, G. Wu, L. Ma, R. Liu, W. Zhong, Z. Luo and X. Fan, “Multi-interactive feature learning and a full-time multi-modality benchmark for image fusion and segmentation,” in Proc. IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 8115–8124. CVF arXiv
- Y. Zhang, Y. Liu, P. Sun, H. Yan, X. Zhao and L. Zhang, “IFCNN: A general image fusion framework based on convolutional neural network,” Information Fusion, vol. 54, pp. 99–118, 2020. doi
- H. Li and X.-J. Wu, “DenseFuse: A fusion approach to infrared and visible images,” IEEE Transactions on Image Processing, vol. 28, no. 5, pp. 2614–2623, 2019. doi arXiv
- K. R. Prabhakar, V. S. Srikar and R. V. Babu, “DeepFuse: A deep unsupervised approach for exposure fusion with extreme exposure image pairs,” in Proc. IEEE International Conference on Computer Vision (ICCV), 2017, pp. 4724–4732. doi CVF
- H. Xu, J. Ma, J. Jiang, X. Guo and H. Ling, “U2Fusion: A unified unsupervised image fusion network,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 1, pp. 502–518, 2022. doi RoadScene
- L. Tang, J. Yuan and J. Ma, “Image fusion in the loop of high-level vision tasks: A semantic-aware real-time infrared and visible image fusion network,” Information Fusion, vol. 82, pp. 28–42, 2022. doi
- J. Ma, L. Tang, F. Fan, J. Huang, X. Mei and Y. Ma, “SwinFusion: Cross-domain long-range learning for general image fusion via Swin Transformer,” IEEE/CAA Journal of Automatica Sinica, vol. 9, no. 7, pp. 1200–1217, 2022. doi
- X. Zhang and Y. Demiris, “Visible and infrared image fusion using deep learning,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 8, pp. 10535–10554, 2023. doi
- Y. Liu, X. Chen, H. Peng and Z. Wang, “Multi-focus image fusion with a deep convolutional neural network,” Information Fusion, vol. 36, pp. 191–207, 2017. doi
- K. Ma, Z. Duanmu, H. Zhu, Y. Fang and Z. Wang, “Deep guided learning for fast multi-exposure image fusion,” IEEE Transactions on Image Processing, vol. 29, pp. 2808–2819, 2020. doi
- Z. Zhao, H. Bai, J. Zhang, Y. Zhang, S. Xu, Z. Lin, R. Timofte and L. Van Gool, “CDDFuse: Correlation-driven dual-branch feature decomposition for multi-modality image fusion,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 5906–5916. CVF arXiv
- J. Ma, W. Yu, P. Liang, C. Li and J. Jiang, “FusionGAN: A generative adversarial network for infrared and visible image fusion,” Information Fusion, vol. 48, pp. 11–26, 2019. doi · public article transcription
- H. Bai, J. Zhang, Z. Zhao, Y. Wu, L. Deng, Y. Cui, T. Feng and S. Xu, “Task-driven image fusion with learnable fusion loss,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025, pp. 7457–7468. CVF arXiv
- Z. Zhao, H. Bai, Y. Zhu, J. Zhang, S. Xu, Y. Zhang, K. Zhang, D. Meng, R. Timofte and L. Van Gool, “DDFM: Denoising diffusion model for multi-modality image fusion,” in Proc. IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 8082–8093. CVF arXiv
- X. Yi, H. Xu, H. Zhang, L. Tang and J. Ma, “Text-IF: Leveraging semantic text guidance for degradation-aware and interactive image fusion,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 27026–27035. CVF arXiv
- D. Wang, J. Liu, X. Fan and R. Liu, “Unsupervised misaligned infrared and visible image fusion via cross-modality image generation and registration,” in Proc. International Joint Conference on Artificial Intelligence (IJCAI), 2022, pp. 3508–3515. doi arXiv
- L. Tang, Y. Deng, Y. Ma, J. Huang and J. Ma, “SuperFusion: A versatile image registration and fusion network with semantic awareness,” IEEE/CAA Journal of Automatica Sinica, vol. 9, no. 12, pp. 2121–2137, 2022. doi
- A. Toet, “The TNO multiband image data collection,” Data in Brief, vol. 15, pp. 249–251, 2017. doi
- L. Tang, J. Yuan, H. Zhang, X. Jiang and J. Ma, “PIAFusion: A progressive infrared and visible image fusion network based on illumination aware,” Information Fusion, vol. 83–84, pp. 79–92, 2022. doi MSRS
- Q. Ha, K. Watanabe, T. Karasawa, Y. Ushiku and T. Harada, “MFNet: Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes,” in Proc. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2017, pp. 5108–5115. doi
- X. Jia, C. Zhu, M. Li, W. Tang and W. Zhou, “LLVIP: A visible-infrared paired dataset for low-light vision,” in Proc. IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, 2021, pp. 3496–3504. CVF arXiv
- J. Cai, S. Gu and L. Zhang, “Learning a deep single image contrast enhancer from multi-exposure images,” IEEE Transactions on Image Processing, vol. 27, no. 4, pp. 2049–2062, 2018. doi
- X. Zhang, “Deep learning-based multi-focus image fusion: A survey and a comparative study,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 9, 2022. doi
- F. Xu, J. Liu, Y. Song, H. Sun and X. Wang, “Multi-exposure image fusion techniques: A comprehensive review,” Remote Sensing, vol. 14, no. 3, art. 771, 2022. doi
- Z. Zhao, H. Bai, J. Zhang, Y. Zhang, K. Zhang, S. Xu, D. Chen, R. Timofte and L. Van Gool, “Equivariant multi-modality image fusion,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 25912–25921. CVF arXiv
- L. Tang, Y. Wang, M. Gong, Z. Li, Y. Deng, X. Yi, C. Li, H. Xu, H. Zhang and J. Ma, “VideoFusion: A spatio-temporal collaborative network for multi-modal video fusion,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026, pp. 19559–19569. CVF
- N. Yokoya, C. Grohnfeldt and J. Chanussot, “Hyperspectral and multispectral data fusion: A comparative review of the recent literature,” IEEE Geoscience and Remote Sensing Magazine, 2017. doi
- H. R. Sheikh and A. C. Bovik, “Image information and visual quality,” IEEE Transactions on Image Processing, 2006. doi
- H. Xu et al., “U2Fusion,” official implementation accompanying the TPAMI paper, 2022. code
- K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” ICLR, 2015. arXiv
- J. Kirkpatrick et al., “Overcoming catastrophic forgetting in neural networks,” Proceedings of the National Academy of Sciences, 2017. doi
- Y. Zhang et al., “IFCNN,” official implementation and errata accompanying the Information Fusion paper, 2020. code
- K. He, X. Zhang, S. Ren and J. Sun, “Deep residual learning for image recognition,” CVPR, 2016. CVF
- J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li and L. Fei-Fei, “ImageNet: A large-scale hierarchical image database,” CVPR, 2009. doi
- N. Silberman, D. Hoiem, P. Kohli and R. Fergus, “Indoor segmentation and support inference from RGBD images,” ECCV, 2012. project
- P. Dhariwal and A. Nichol, “Diffusion models beat GANs on image synthesis,” NeurIPS, 2021. arXiv
- J. Liu et al., “TarDAL,” official implementation accompanying the CVPR paper, 2022. code
- G. Jocher et al., “YOLOv5,” Ultralytics object-detection software, 2020 onward. code
- J. Ma et al., “FusionGAN,” official implementation accompanying the Information Fusion paper, 2019. code
- L. Tang, J. Yuan and J. Ma, “SeAFusion,” official implementation accompanying the Information Fusion paper, 2022. code
- T. Mertens, J. Kautz and F. Van Reeth, “Exposure Fusion,” in Proc. Pacific Graphics, 2007. paper · author project