Full search page →

TarDAL: fuse for both appearance and object detection

Optional · paper guide 11 of 11 for DLCV 10 · Image Fusion: Complementary Images, Useful Outputs. Citation numbers refer to the reference list at the bottom of this page.

Navigate: this page and this track: Navigate: this page and this track

On this page

Supplementary notes · 6

  1. The defocus spread effect
  2. Building and collapsing a Laplacian pyramid
  3. Sparse-representation fusion
  4. Mutual information as a fusion metric
  5. Training signals of representative fusion networks
  6. VIF and source-referenced fusion scores

Paper guides · 11

  1. Exposure fusion (Mertens, Kautz and Van Reeth)
  2. VIFB: a visible and infrared image fusion benchmark
  3. The infrared–visible fusion survey of Liu et al. (TPAMI 2025)
  4. DenseFuse: learn reconstruction, choose fusion in feature space
  5. DDFM: a diffusion prior with likelihood rectification
  6. FusionGAN: thermal intensity with an adversarial visible-image prior
  7. IFCNN: supervised fusion with a variable number of inputs
  8. U2Fusion: one unsupervised model across fusion tasks
  9. SwinFusion: attention within and across the sources
  10. SeAFusion: let semantic segmentation influence the fused image
  11. TarDAL: fuse for both appearance and object detection

Deep Learning for Computer Vision

  1. 02 · Deep Image Classification
View the whole course tree →

Liu et al.’s CVPR 2022 paper makes downstream detection part of the fusion objective [27]. A generator preserves common structure, two discriminators specialize in thermal targets and visible detail, and cooperative training lets detection gradients influence the fused image.

Challenge

A fused image can score well on entropy or look attractive while making an object harder to detect. Conversely, optimizing only a detector can produce an intermediate image that is poor for human inspection. TarDAL asks for both visual usefulness and detection performance.

The two modalities also contribute differently. Infrared can make a warm foreground target salient, while visible images provide background texture. A single discriminator that treats all pixels identically may not express those preferences. TarDAL therefore separates the adversarial comparisons by content and representation.

Method

Separate the coupled fusion/detection objective from target/detail adversarial branches; detector supervision changes what information the fusion model is rewarded for retaining.
Original paper figure 2 — Separate the coupled fusion/detection objective from target/detail adversarial branches; detector supervision changes what information the fusion model is rewarded for retaining.

Figure source: [27], Published paper PDF, PDF page 3. Original PDF.

Fusion and detection together. Let xx be infrared, yy visible, u=Gθ(x,y)u=G_\theta(x,y) the fused image, and Ψη\Psi_\eta a detector. The operational joint objective has the form

min⁡θ,ηLd(Ψη(Gθ(x,y)),a)+λLf(Gθ(x,y);x,y).\min_{\theta,\eta} \mathcal L_d\bigl(\Psi_\eta(G_\theta(x,y)),a\bigr) +\lambda\mathcal L_f\bigl(G_\theta(x,y);x,y\bigr).

aa contains annotated boxes/classes, θ\theta the fusion parameters, η\eta the detector parameters, Ld\mathcal L_d the detection loss, Lf\mathcal L_f a fusion regularizer and λ\lambda their trade-off. This is the trainable form of the paper’s bilevel motivation. It is not a fully solved nested optimization at every update.

The detector uses YOLOv5 [65]. A detection gradient reaches the fusion parameters through the detector and generator by the chain rule. The fusion regularizer supplies an additional gradient that prevents the image from becoming merely an arbitrary detector-friendly code. Detection labels therefore enter cooperative training even though a correct fused target image is unavailable.

Generator. The paper uses a five-layer dense block for feature extraction and a three-convolution merging block. Source features share the same spatial grid, and the output retains the source image size. Dense connections reuse intermediate information; the reconstruction path combines it into one image. No discriminator is needed to produce an image after training.

Target mask. A pretrained saliency network estimates foreground thermal targets from the infrared image. A mask m∈[0,1]H×Wm\in[0,1]^{H\times W} emphasizes these targets; its complement emphasizes background. This is a learned proxy for target importance, not a perfect object annotation. Errors in the mask alter which evidence receives adversarial supervision.

Two discriminators. The target critic DTD_T compares m⊙xm\odot x with m⊙um\odot u, encouraging thermal-target intensity preservation. The detail critic DDD_D compares visible and fused gradient representations, optionally restricted to the background. ⊙\odot denotes element-wise multiplication and a spatial gradient operator supplies edge magnitude. The critics share an architectural pattern but have separate parameters and inputs.

Their adversarial training uses real/fused score differences with a gradient-norm regularizer. In a consistent critic-maximization convention, its schematic form is E[D(real)]−E[D(fake)]−kE∥∇D∥p\mathbb E[D(\mathrm{real})]-\mathbb E[D(\mathrm{fake})]-k\mathbb E\|\nabla D\|^p. The paper uses k=2,p=6k=2,p=6 [27]. The generator instead seeks high critic scores for its fused target and detail representations. These objectives must retain consistent signs when translated into minimizing code.

Source reconstruction. Structural similarity constrains common structure, and saliency-dependent pixel fidelity balances the sources. The published objective combines these with adversarial supervision as Lf=LSSIM+αLpixel+βLadv\mathcal L_f=\mathcal L_{\mathrm{SSIM}}+\alpha\mathcal L_{\mathrm{pixel}}+\beta\mathcal L_{\mathrm{adv}}, with α,β\alpha,\beta weighting the components [27].

There is an important reproduction caveat. The PDF’s Eq. 5 prints a saliency denominator containing Sx−SyS_x-S_y, which is not a valid nonnegative normalization in general. The current official training code does not implement that expression literally: one path computes w1=0.5+0.5(s1−s2)w_1=0.5+0.5(s_1-s_2) and w2=1−w1w_2=1-w_1, then weights per-source structural and L1 losses [64]. Here s1,s2s_1,s_2 are the supplied saliency estimates; the weights’ nonnegativity depends on their scale. Other current configuration paths offer different source-loss options. Match the code revision and configuration instead of silently treating the typeset expression, current code and published experiment as identical.

Cooperative updates. Training alternates critic updates with fusion/detection learning. The detector learns from fused images, while the generator receives both fusion and detection supervision. This differs from first training a fusion model independently and subsequently training a detector on its fixed outputs. At deployment, the fused image can be used alone for viewing or passed to the detector; its reported cost should state which path is timed.

Results

The paper introduces M3FD with 4,200 aligned infrared–visible pairs for detection, covering daytime, overcast, night and challenging conditions, plus a separate fusion comparison collection [27]. Its acquisition system and registration are part of the dataset contribution; the model does not automatically align arbitrary camera pairs.

Table 2 reports detection with the stated YOLOv5 setup. TarDAL gives mAP@0.5 0.613 on the Multispectral dataset and 0.807 on M3FD [27]. The visible-only and infrared-only M3FD rows give 0.756 and 0.781. These are results under the paper’s detector and data protocol, not a detector-independent quality score for fusion.

The same table lists the fusion model at 0.296 million parameters, 14.88 GFLOPs and 0.041 seconds in its evaluation setup [27]. Keep the timing boundary, resolution and implementation attached; this number does not include an unspecified complete camera-to-detection system.

Ablations

Table 3 separates the critics. With neither critic, M3FD mAP@0.5 is 0.722; with only the detail critic it is 0.719; with only the target critic 0.781; with both 0.807 [27]. The detail critic’s role in visual texture does not imply that it always raises detection AP when added alone.

This same table demonstrates metric disagreement. On TNO, the no-critic row has EN 7.223 / SD 53.107, above the full model’s 7.177 / 51.352, even though the paper shows noisier outputs without the critics [27]. Higher entropy and spread are not sufficient evidence of better fusion.

Table 4 compares training strategies. Direct training gives M3FD 0.807, task-oriented training 0.809, and cooperative training 0.811 mAP@0.5. On the Multispectral dataset the corresponding values are 0.613,0.615,0.624 [27]. Task-only training can improve detection while harming appearance; the cooperative objective seeks a better compromise. Figure 10 additionally examines saliency weighting and masks visually rather than providing a complete numerical isolation of each term.

Conclusion

TarDAL specifies usefulness through a downstream task while retaining explicit visual constraints. The dual critics encode different modality contributions, and cooperative training lets the generator adapt to detection rather than optimize only generic image statistics.

Problems

The fusion is tied to training labels, saliency estimation, a particular detector and a particular data distribution. Better AP for that detector does not guarantee transfer to another architecture or task. The published tables and current code also require careful version matching.

Finally, a displayed fused image is not a thermal measurement. Learned intensity changes can help recognition while altering quantitative sensor interpretation. Evaluate visual artifacts, detection errors, registration failures and efficiency separately; a single winning metric cannot certify them all.

References

  1. Y. Liu, X. Chen, Z. Wang, Z. J. Wang, R. K. Ward and X. Wang, “Deep learning for pixel-level image fusion: Recent advances and future prospects,” Information Fusion, vol. 42, pp. 158–173, 2018. doi
  2. J. Liu, G. Wu, Z. Liu, D. Wang, Z. Jiang, L. Ma, W. Zhong, X. Fan and R. Liu, “Infrared and visible image fusion: From data compatibility to task adaption,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 47, no. 4, pp. 2349–2369, 2025. doi arXiv
  3. J. Ma, Y. Ma and C. Li, “Infrared and visible image fusion methods and applications: A survey,” Information Fusion, vol. 45, pp. 153–178, 2019. doi
  4. P. E. Debevec and J. Malik, “Recovering high dynamic range radiance maps from photographs,” in Proc. SIGGRAPH, 1997, pp. 369–378. doi
  5. T. Mertens, J. Kautz and F. Van Reeth, “Exposure fusion: A simple and practical alternative to high dynamic range photography,” Computer Graphics Forum, vol. 28, no. 1, pp. 161–171, 2009. doi
  6. S. Xu, X. Wei, C. Zhang, J. Liu and J. Zhang, “MFFW: A new dataset for multi-focus image fusion,” arXiv:2002.04780, 2020. arXiv
  7. X. Zhang, “Multi-focus image fusion: A benchmark,” arXiv:2005.01116, 2020. arXiv
  8. P. J. Burt and E. H. Adelson, “The Laplacian pyramid as a compact image code,” IEEE Transactions on Communications, vol. 31, no. 4, pp. 532–540, 1983. doi
  9. P. J. Burt and R. J. Kolczynski, “Enhanced image capture through fusion,” in Proc. 4th International Conference on Computer Vision (ICCV), 1993, pp. 173–182. doi
  10. P. J. Burt and E. H. Adelson, “A multiresolution spline with application to image mosaics,” ACM Transactions on Graphics, vol. 2, no. 4, pp. 217–236, 1983. doi
  11. H. Li, B. S. Manjunath and S. K. Mitra, “Multisensor image fusion using the wavelet transform,” Graphical Models and Image Processing, vol. 57, no. 3, pp. 235–245, 1995. doi
  12. S. Li, X. Kang and J. Hu, “Image fusion with guided filtering,” IEEE Transactions on Image Processing, vol. 22, no. 7, pp. 2864–2875, 2013. doi
  13. K. He, J. Sun and X. Tang, “Guided image filtering,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, no. 6, pp. 1397–1409, 2013. doi
  14. B. Yang and S. Li, “Multifocus image fusion and restoration with sparse representation,” IEEE Transactions on Instrumentation and Measurement, vol. 59, no. 4, pp. 884–892, 2010. doi
  15. M. Nejati, S. Samavi and S. Shirani, “Multi-focus image fusion using dictionary-based sparse representation,” Information Fusion, vol. 25, pp. 72–84, 2015. doi
  16. Z. Wang, A. C. Bovik, H. R. Sheikh and E. P. Simoncelli, “Image quality assessment: From error visibility to structural similarity,” IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600–612, 2004. doi
  17. X. Zhang, P. Ye and G. Xiao, “VIFB: A visible and infrared image fusion benchmark,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2020. CVF arXiv
  18. X. Zhang, “Benchmarking and comparing multi-exposure image fusion algorithms,” Information Fusion, vol. 74, pp. 111–131, 2021. doi arXiv
  19. A. M. Eskicioglu and P. S. Fisher, “Image quality measures and their performance,” IEEE Transactions on Communications, vol. 43, no. 12, pp. 2959–2965, 1995. doi
  20. G. Qu, D. Zhang and P. Yan, “Information measure for performance of image fusion,” Electronics Letters, vol. 38, no. 7, pp. 313–315, 2002. doi
  21. M. Hossny, S. Nahavandi and D. Creighton, “Comments on ‘Information measure for performance of image fusion’,” Electronics Letters, vol. 44, no. 18, pp. 1066–1067, 2008. doi
  22. C. S. Xydeas and V. Petrović, “Objective image fusion performance measure,” Electronics Letters, vol. 36, no. 4, pp. 308–309, 2000. doi
  23. G. Piella and H. Heijmans, “A new quality metric for image fusion,” in Proc. IEEE International Conference on Image Processing (ICIP), 2003, vol. 3, pp. III-173–176. doi
  24. Y. Han, Y. Cai, Y. Cao and X. Xu, “A new image fusion performance metric based on visual information fidelity,” Information Fusion, vol. 14, no. 2, pp. 127–135, 2013. doi
  25. K. Ma, K. Zeng and Z. Wang, “Perceptual quality assessment for multi-exposure image fusion,” IEEE Transactions on Image Processing, vol. 24, no. 11, pp. 3345–3356, 2015. doi
  26. Z. Liu, E. Blasch, Z. Xue, J. Zhao, R. Laganiere and W. Wu, “Objective assessment of multiresolution image fusion algorithms for context enhancement in night vision: A comparative study,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 34, no. 1, pp. 94–109, 2012. doi
  27. J. Liu, X. Fan, Z. Huang, G. Wu, R. Liu, W. Zhong and Z. Luo, “Target-aware dual adversarial learning and a multi-scenario multi-modality benchmark to fuse infrared and visible for object detection,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 5802–5811. CVF arXiv M3FD
  28. J. Liu, Z. Liu, G. Wu, L. Ma, R. Liu, W. Zhong, Z. Luo and X. Fan, “Multi-interactive feature learning and a full-time multi-modality benchmark for image fusion and segmentation,” in Proc. IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 8115–8124. CVF arXiv
  29. Y. Zhang, Y. Liu, P. Sun, H. Yan, X. Zhao and L. Zhang, “IFCNN: A general image fusion framework based on convolutional neural network,” Information Fusion, vol. 54, pp. 99–118, 2020. doi
  30. H. Li and X.-J. Wu, “DenseFuse: A fusion approach to infrared and visible images,” IEEE Transactions on Image Processing, vol. 28, no. 5, pp. 2614–2623, 2019. doi arXiv
  31. K. R. Prabhakar, V. S. Srikar and R. V. Babu, “DeepFuse: A deep unsupervised approach for exposure fusion with extreme exposure image pairs,” in Proc. IEEE International Conference on Computer Vision (ICCV), 2017, pp. 4724–4732. doi CVF
  32. H. Xu, J. Ma, J. Jiang, X. Guo and H. Ling, “U2Fusion: A unified unsupervised image fusion network,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 1, pp. 502–518, 2022. doi RoadScene
  33. L. Tang, J. Yuan and J. Ma, “Image fusion in the loop of high-level vision tasks: A semantic-aware real-time infrared and visible image fusion network,” Information Fusion, vol. 82, pp. 28–42, 2022. doi
  34. J. Ma, L. Tang, F. Fan, J. Huang, X. Mei and Y. Ma, “SwinFusion: Cross-domain long-range learning for general image fusion via Swin Transformer,” IEEE/CAA Journal of Automatica Sinica, vol. 9, no. 7, pp. 1200–1217, 2022. doi
  35. X. Zhang and Y. Demiris, “Visible and infrared image fusion using deep learning,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 8, pp. 10535–10554, 2023. doi
  36. Y. Liu, X. Chen, H. Peng and Z. Wang, “Multi-focus image fusion with a deep convolutional neural network,” Information Fusion, vol. 36, pp. 191–207, 2017. doi
  37. K. Ma, Z. Duanmu, H. Zhu, Y. Fang and Z. Wang, “Deep guided learning for fast multi-exposure image fusion,” IEEE Transactions on Image Processing, vol. 29, pp. 2808–2819, 2020. doi
  38. Z. Zhao, H. Bai, J. Zhang, Y. Zhang, S. Xu, Z. Lin, R. Timofte and L. Van Gool, “CDDFuse: Correlation-driven dual-branch feature decomposition for multi-modality image fusion,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 5906–5916. CVF arXiv
  39. J. Ma, W. Yu, P. Liang, C. Li and J. Jiang, “FusionGAN: A generative adversarial network for infrared and visible image fusion,” Information Fusion, vol. 48, pp. 11–26, 2019. doi · public article transcription
  40. H. Bai, J. Zhang, Z. Zhao, Y. Wu, L. Deng, Y. Cui, T. Feng and S. Xu, “Task-driven image fusion with learnable fusion loss,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025, pp. 7457–7468. CVF arXiv
  41. Z. Zhao, H. Bai, Y. Zhu, J. Zhang, S. Xu, Y. Zhang, K. Zhang, D. Meng, R. Timofte and L. Van Gool, “DDFM: Denoising diffusion model for multi-modality image fusion,” in Proc. IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 8082–8093. CVF arXiv
  42. X. Yi, H. Xu, H. Zhang, L. Tang and J. Ma, “Text-IF: Leveraging semantic text guidance for degradation-aware and interactive image fusion,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 27026–27035. CVF arXiv
  43. D. Wang, J. Liu, X. Fan and R. Liu, “Unsupervised misaligned infrared and visible image fusion via cross-modality image generation and registration,” in Proc. International Joint Conference on Artificial Intelligence (IJCAI), 2022, pp. 3508–3515. doi arXiv
  44. L. Tang, Y. Deng, Y. Ma, J. Huang and J. Ma, “SuperFusion: A versatile image registration and fusion network with semantic awareness,” IEEE/CAA Journal of Automatica Sinica, vol. 9, no. 12, pp. 2121–2137, 2022. doi
  45. A. Toet, “The TNO multiband image data collection,” Data in Brief, vol. 15, pp. 249–251, 2017. doi
  46. L. Tang, J. Yuan, H. Zhang, X. Jiang and J. Ma, “PIAFusion: A progressive infrared and visible image fusion network based on illumination aware,” Information Fusion, vol. 83–84, pp. 79–92, 2022. doi MSRS
  47. Q. Ha, K. Watanabe, T. Karasawa, Y. Ushiku and T. Harada, “MFNet: Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes,” in Proc. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2017, pp. 5108–5115. doi
  48. X. Jia, C. Zhu, M. Li, W. Tang and W. Zhou, “LLVIP: A visible-infrared paired dataset for low-light vision,” in Proc. IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, 2021, pp. 3496–3504. CVF arXiv
  49. J. Cai, S. Gu and L. Zhang, “Learning a deep single image contrast enhancer from multi-exposure images,” IEEE Transactions on Image Processing, vol. 27, no. 4, pp. 2049–2062, 2018. doi
  50. X. Zhang, “Deep learning-based multi-focus image fusion: A survey and a comparative study,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 9, 2022. doi
  51. F. Xu, J. Liu, Y. Song, H. Sun and X. Wang, “Multi-exposure image fusion techniques: A comprehensive review,” Remote Sensing, vol. 14, no. 3, art. 771, 2022. doi
  52. Z. Zhao, H. Bai, J. Zhang, Y. Zhang, K. Zhang, S. Xu, D. Chen, R. Timofte and L. Van Gool, “Equivariant multi-modality image fusion,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 25912–25921. CVF arXiv
  53. L. Tang, Y. Wang, M. Gong, Z. Li, Y. Deng, X. Yi, C. Li, H. Xu, H. Zhang and J. Ma, “VideoFusion: A spatio-temporal collaborative network for multi-modal video fusion,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026, pp. 19559–19569. CVF
  54. N. Yokoya, C. Grohnfeldt and J. Chanussot, “Hyperspectral and multispectral data fusion: A comparative review of the recent literature,” IEEE Geoscience and Remote Sensing Magazine, 2017. doi
  55. H. R. Sheikh and A. C. Bovik, “Image information and visual quality,” IEEE Transactions on Image Processing, 2006. doi
  56. H. Xu et al., “U2Fusion,” official implementation accompanying the TPAMI paper, 2022. code
  57. K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” ICLR, 2015. arXiv
  58. J. Kirkpatrick et al., “Overcoming catastrophic forgetting in neural networks,” Proceedings of the National Academy of Sciences, 2017. doi
  59. Y. Zhang et al., “IFCNN,” official implementation and errata accompanying the Information Fusion paper, 2020. code
  60. K. He, X. Zhang, S. Ren and J. Sun, “Deep residual learning for image recognition,” CVPR, 2016. CVF
  61. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li and L. Fei-Fei, “ImageNet: A large-scale hierarchical image database,” CVPR, 2009. doi
  62. N. Silberman, D. Hoiem, P. Kohli and R. Fergus, “Indoor segmentation and support inference from RGBD images,” ECCV, 2012. project
  63. P. Dhariwal and A. Nichol, “Diffusion models beat GANs on image synthesis,” NeurIPS, 2021. arXiv
  64. J. Liu et al., “TarDAL,” official implementation accompanying the CVPR paper, 2022. code
  65. G. Jocher et al., “YOLOv5,” Ultralytics object-detection software, 2020 onward. code
  66. J. Ma et al., “FusionGAN,” official implementation accompanying the Information Fusion paper, 2019. code
  67. L. Tang, J. Yuan and J. Ma, “SeAFusion,” official implementation accompanying the Information Fusion paper, 2022. code
  68. T. Mertens, J. Kautz and F. Van Reeth, “Exposure Fusion,” in Proc. Pacific Graphics, 2007. paper · author project