Full search page →

FusionGAN: thermal intensity with an adversarial visible-image prior

Optional · paper guide 6 of 11 for DLCV 10 · Image Fusion: Complementary Images, Useful Outputs. Citation numbers refer to the reference list at the bottom of this page.

Navigate: this page and this track: Navigate: this page and this track

On this page

Supplementary notes · 6

  1. The defocus spread effect
  2. Building and collapsing a Laplacian pyramid
  3. Sparse-representation fusion
  4. Mutual information as a fusion metric
  5. Training signals of representative fusion networks
  6. VIF and source-referenced fusion scores

Paper guides · 11

  1. Exposure fusion (Mertens, Kautz and Van Reeth)
  2. VIFB: a visible and infrared image fusion benchmark
  3. The infrared–visible fusion survey of Liu et al. (TPAMI 2025)
  4. DenseFuse: learn reconstruction, choose fusion in feature space
  5. DDFM: a diffusion prior with likelihood rectification
  6. FusionGAN: thermal intensity with an adversarial visible-image prior
  7. IFCNN: supervised fusion with a variable number of inputs
  8. U2Fusion: one unsupervised model across fusion tasks
  9. SwinFusion: attention within and across the sources
  10. SeAFusion: let semantic segmentation influence the fused image
  11. TarDAL: fuse for both appearance and object detection

Deep Learning for Computer Vision

  1. 02 · Deep Image Classification
View the whole course tree →

Challenge

An infrared image often makes a thermally contrasting target conspicuous, while a visible image carries background texture. Averaging can weaken both. FusionGAN [39] asks whether a learned image distribution can supplement the source-preservation objective. Its discriminator treats visible images as real and generated fused images as fake; the generator must also remain close to infrared intensity and visible gradients. This makes the two source roles deliberately asymmetric.

The original article [39] establishes the experimental evidence; the released implementation [66] supplies the code settings identified below.

Method

Teaching training/inference diagram — content and adversarial terms update the generator during training; inference runs the trained generator without the discriminator or losses.
Teaching training/inference diagram — content and adversarial terms update the generator during training; inference runs the trained generator without the discriminator or losses.

A paired generator combines spatially corresponding inputs. The released TensorFlow generator concatenates the infrared and visible grayscale patches along the channel axis. For batch size BB and patch dimensions H,WH,W, the input is B×H×W×2B\times H\times W\times2. Its five stride-one convolutions have kernel widths 5, 5, 3, 3 and 1, with output channels 256, 128, 64, 32 and 1. The first four have batch normalization and LeakyReLU; the final layer uses tanh. Spectral normalization is applied to the learned convolution weights in this implementation.

The code uses valid convolutions, so the spatial output is (H−12)×(W−12)(H-12)\times(W-12). Training labels and inference padding must respect this reduction. A generic same-padding reimplementation with identical channel counts is not exactly this network. Pixel intensities are normalized for the tanh range rather than interpreted directly as eight-bit values.

The discriminator supplies a visible-image appearance criterion. Four stride-two 3 × 3 convolutions produce 32, 64, 128 and 256 channels; the flattened feature feeds a scalar linear output. Real examples are visible patches, not nonexistent “correct fused” targets. A high discriminator score therefore means compatibility with the trained visible-image distribution, not proof that a thermal target was preserved.

Writing F=G(I,V)F=G(I,V) for the generated image, the released generator objective has the form

LG=E[(D(F)−a)2]+100[MSE⁡(F,I)+5MSE⁡(ΔF,ΔV)].\mathcal L_G=\mathbb E[(D(F)-a)^2]+100\left[\operatorname{MSE}(F,I)+5\operatorname{MSE}(\Delta F,\Delta V)\right].

Here II is the infrared label patch, VV is the visible label patch, GG is the fusion network, DD is the scalar discriminator, aa is a real-like training label, Δ\Delta is the four-neighbor discrete Laplacian used by the released gradient helper, and MSE averages squared elementwise differences. Inspect that helper when reproducing the objective: a generic Sobel implementation is not automatically identical. The outer expectation averages the adversarial residual over training examples and sampled labels.

The intensity term protects thermal evidence. The spatial term protects visible detail. The adversarial term penalizes fused appearances that a visible-image discriminator can easily reject. Their weights are an engineering compromise: an infrared pixel and a visible pixel need not have the same radiometric meaning, and no coefficient makes them a calibrated physical target.

The discriminator minimizes squared residuals toward real-like labels for visible patches and fake-like labels for fused patches. In the released code, real-like labels are sampled uniformly from 0.7 to 1.2, and fake-like labels from 0 to 0.3. Those intervals are label-smoothing settings, not confidence bounds or benchmark scores. The generator aims at a sampled real-like label. Consequently this code is better described by its explicit least-squares objectives than by copying a generic logarithmic GAN minimax equation.

Optimization alternates the roles. The training loop updates the discriminator twice, then the generator once for each batch, using Adam. The discriminator must learn from the generator’s current outputs; the generator must receive gradients through the discriminator while optimizing its own parameters. At inference only the generator is required. There is no random latent input in this paired generator, so the GAN terminology does not imply stochastic multi-sample fusion for a fixed input pair.

Results

The original paper [39, Table 1] reports mean inference time in seconds:

Test setFusionGAN (GPU)
TNO10.254
TNO20.0716
INO0.0350

The eight comparison methods ran on CPU. These are mixed-hardware timings, not a controlled speed ranking. Sections 4.2.2–4.2.3 also report trade-offs: FusionGAN does not lead every source-based metric. Check thermal targets and visible textures separately rather than treating a single score as overall fidelity.

Ablations

Section 4.3 and Figure 10 train with and without adversarial loss while keeping the other settings fixed. On four TNO pairs, the adversarial version preserves more visible detail and contrast. This is a qualitative comparison; no numerical ablation delta is tabulated. It supports the appearance contribution of the discriminator, not a demonstrated detection gain. Removing intensity or spatial terms would be additional experiments, not results established by this comparison.

Conclusion

FusionGAN makes the fusion objective asymmetric and adds an adversarial appearance constraint without requiring a fused target image. Understanding the source role of every term is more useful than the general label “GAN-based”: the discriminator favors visible appearance, while the infrared content term must preserve the complementary thermal evidence.

Problems

A visible-image discriminator can oppose unusual thermal structures; reducing its loss is not the same as improving a detection task. Misalignment makes the intensity and spatial objectives disagree at boundaries. Noisy visible detail can be retained as if it were meaningful texture, and a deterministic generator cannot express uncertainty about conflicting evidence.

Batch normalization, valid-convolution boundaries and historical TensorFlow operations also matter for faithful reproduction. Compare TarDAL for separate target/detail critics and a cooperative detector, and U2Fusion for a non-adversarial, information-adaptive objective.

References

  1. Y. Liu, X. Chen, Z. Wang, Z. J. Wang, R. K. Ward and X. Wang, “Deep learning for pixel-level image fusion: Recent advances and future prospects,” Information Fusion, vol. 42, pp. 158–173, 2018. doi
  2. J. Liu, G. Wu, Z. Liu, D. Wang, Z. Jiang, L. Ma, W. Zhong, X. Fan and R. Liu, “Infrared and visible image fusion: From data compatibility to task adaption,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 47, no. 4, pp. 2349–2369, 2025. doi arXiv
  3. J. Ma, Y. Ma and C. Li, “Infrared and visible image fusion methods and applications: A survey,” Information Fusion, vol. 45, pp. 153–178, 2019. doi
  4. P. E. Debevec and J. Malik, “Recovering high dynamic range radiance maps from photographs,” in Proc. SIGGRAPH, 1997, pp. 369–378. doi
  5. T. Mertens, J. Kautz and F. Van Reeth, “Exposure fusion: A simple and practical alternative to high dynamic range photography,” Computer Graphics Forum, vol. 28, no. 1, pp. 161–171, 2009. doi
  6. S. Xu, X. Wei, C. Zhang, J. Liu and J. Zhang, “MFFW: A new dataset for multi-focus image fusion,” arXiv:2002.04780, 2020. arXiv
  7. X. Zhang, “Multi-focus image fusion: A benchmark,” arXiv:2005.01116, 2020. arXiv
  8. P. J. Burt and E. H. Adelson, “The Laplacian pyramid as a compact image code,” IEEE Transactions on Communications, vol. 31, no. 4, pp. 532–540, 1983. doi
  9. P. J. Burt and R. J. Kolczynski, “Enhanced image capture through fusion,” in Proc. 4th International Conference on Computer Vision (ICCV), 1993, pp. 173–182. doi
  10. P. J. Burt and E. H. Adelson, “A multiresolution spline with application to image mosaics,” ACM Transactions on Graphics, vol. 2, no. 4, pp. 217–236, 1983. doi
  11. H. Li, B. S. Manjunath and S. K. Mitra, “Multisensor image fusion using the wavelet transform,” Graphical Models and Image Processing, vol. 57, no. 3, pp. 235–245, 1995. doi
  12. S. Li, X. Kang and J. Hu, “Image fusion with guided filtering,” IEEE Transactions on Image Processing, vol. 22, no. 7, pp. 2864–2875, 2013. doi
  13. K. He, J. Sun and X. Tang, “Guided image filtering,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, no. 6, pp. 1397–1409, 2013. doi
  14. B. Yang and S. Li, “Multifocus image fusion and restoration with sparse representation,” IEEE Transactions on Instrumentation and Measurement, vol. 59, no. 4, pp. 884–892, 2010. doi
  15. M. Nejati, S. Samavi and S. Shirani, “Multi-focus image fusion using dictionary-based sparse representation,” Information Fusion, vol. 25, pp. 72–84, 2015. doi
  16. Z. Wang, A. C. Bovik, H. R. Sheikh and E. P. Simoncelli, “Image quality assessment: From error visibility to structural similarity,” IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600–612, 2004. doi
  17. X. Zhang, P. Ye and G. Xiao, “VIFB: A visible and infrared image fusion benchmark,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2020. CVF arXiv
  18. X. Zhang, “Benchmarking and comparing multi-exposure image fusion algorithms,” Information Fusion, vol. 74, pp. 111–131, 2021. doi arXiv
  19. A. M. Eskicioglu and P. S. Fisher, “Image quality measures and their performance,” IEEE Transactions on Communications, vol. 43, no. 12, pp. 2959–2965, 1995. doi
  20. G. Qu, D. Zhang and P. Yan, “Information measure for performance of image fusion,” Electronics Letters, vol. 38, no. 7, pp. 313–315, 2002. doi
  21. M. Hossny, S. Nahavandi and D. Creighton, “Comments on ‘Information measure for performance of image fusion’,” Electronics Letters, vol. 44, no. 18, pp. 1066–1067, 2008. doi
  22. C. S. Xydeas and V. Petrović, “Objective image fusion performance measure,” Electronics Letters, vol. 36, no. 4, pp. 308–309, 2000. doi
  23. G. Piella and H. Heijmans, “A new quality metric for image fusion,” in Proc. IEEE International Conference on Image Processing (ICIP), 2003, vol. 3, pp. III-173–176. doi
  24. Y. Han, Y. Cai, Y. Cao and X. Xu, “A new image fusion performance metric based on visual information fidelity,” Information Fusion, vol. 14, no. 2, pp. 127–135, 2013. doi
  25. K. Ma, K. Zeng and Z. Wang, “Perceptual quality assessment for multi-exposure image fusion,” IEEE Transactions on Image Processing, vol. 24, no. 11, pp. 3345–3356, 2015. doi
  26. Z. Liu, E. Blasch, Z. Xue, J. Zhao, R. Laganiere and W. Wu, “Objective assessment of multiresolution image fusion algorithms for context enhancement in night vision: A comparative study,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 34, no. 1, pp. 94–109, 2012. doi
  27. J. Liu, X. Fan, Z. Huang, G. Wu, R. Liu, W. Zhong and Z. Luo, “Target-aware dual adversarial learning and a multi-scenario multi-modality benchmark to fuse infrared and visible for object detection,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 5802–5811. CVF arXiv M3FD
  28. J. Liu, Z. Liu, G. Wu, L. Ma, R. Liu, W. Zhong, Z. Luo and X. Fan, “Multi-interactive feature learning and a full-time multi-modality benchmark for image fusion and segmentation,” in Proc. IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 8115–8124. CVF arXiv
  29. Y. Zhang, Y. Liu, P. Sun, H. Yan, X. Zhao and L. Zhang, “IFCNN: A general image fusion framework based on convolutional neural network,” Information Fusion, vol. 54, pp. 99–118, 2020. doi
  30. H. Li and X.-J. Wu, “DenseFuse: A fusion approach to infrared and visible images,” IEEE Transactions on Image Processing, vol. 28, no. 5, pp. 2614–2623, 2019. doi arXiv
  31. K. R. Prabhakar, V. S. Srikar and R. V. Babu, “DeepFuse: A deep unsupervised approach for exposure fusion with extreme exposure image pairs,” in Proc. IEEE International Conference on Computer Vision (ICCV), 2017, pp. 4724–4732. doi CVF
  32. H. Xu, J. Ma, J. Jiang, X. Guo and H. Ling, “U2Fusion: A unified unsupervised image fusion network,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 1, pp. 502–518, 2022. doi RoadScene
  33. L. Tang, J. Yuan and J. Ma, “Image fusion in the loop of high-level vision tasks: A semantic-aware real-time infrared and visible image fusion network,” Information Fusion, vol. 82, pp. 28–42, 2022. doi
  34. J. Ma, L. Tang, F. Fan, J. Huang, X. Mei and Y. Ma, “SwinFusion: Cross-domain long-range learning for general image fusion via Swin Transformer,” IEEE/CAA Journal of Automatica Sinica, vol. 9, no. 7, pp. 1200–1217, 2022. doi
  35. X. Zhang and Y. Demiris, “Visible and infrared image fusion using deep learning,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 8, pp. 10535–10554, 2023. doi
  36. Y. Liu, X. Chen, H. Peng and Z. Wang, “Multi-focus image fusion with a deep convolutional neural network,” Information Fusion, vol. 36, pp. 191–207, 2017. doi
  37. K. Ma, Z. Duanmu, H. Zhu, Y. Fang and Z. Wang, “Deep guided learning for fast multi-exposure image fusion,” IEEE Transactions on Image Processing, vol. 29, pp. 2808–2819, 2020. doi
  38. Z. Zhao, H. Bai, J. Zhang, Y. Zhang, S. Xu, Z. Lin, R. Timofte and L. Van Gool, “CDDFuse: Correlation-driven dual-branch feature decomposition for multi-modality image fusion,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 5906–5916. CVF arXiv
  39. J. Ma, W. Yu, P. Liang, C. Li and J. Jiang, “FusionGAN: A generative adversarial network for infrared and visible image fusion,” Information Fusion, vol. 48, pp. 11–26, 2019. doi · public article transcription
  40. H. Bai, J. Zhang, Z. Zhao, Y. Wu, L. Deng, Y. Cui, T. Feng and S. Xu, “Task-driven image fusion with learnable fusion loss,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025, pp. 7457–7468. CVF arXiv
  41. Z. Zhao, H. Bai, Y. Zhu, J. Zhang, S. Xu, Y. Zhang, K. Zhang, D. Meng, R. Timofte and L. Van Gool, “DDFM: Denoising diffusion model for multi-modality image fusion,” in Proc. IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 8082–8093. CVF arXiv
  42. X. Yi, H. Xu, H. Zhang, L. Tang and J. Ma, “Text-IF: Leveraging semantic text guidance for degradation-aware and interactive image fusion,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 27026–27035. CVF arXiv
  43. D. Wang, J. Liu, X. Fan and R. Liu, “Unsupervised misaligned infrared and visible image fusion via cross-modality image generation and registration,” in Proc. International Joint Conference on Artificial Intelligence (IJCAI), 2022, pp. 3508–3515. doi arXiv
  44. L. Tang, Y. Deng, Y. Ma, J. Huang and J. Ma, “SuperFusion: A versatile image registration and fusion network with semantic awareness,” IEEE/CAA Journal of Automatica Sinica, vol. 9, no. 12, pp. 2121–2137, 2022. doi
  45. A. Toet, “The TNO multiband image data collection,” Data in Brief, vol. 15, pp. 249–251, 2017. doi
  46. L. Tang, J. Yuan, H. Zhang, X. Jiang and J. Ma, “PIAFusion: A progressive infrared and visible image fusion network based on illumination aware,” Information Fusion, vol. 83–84, pp. 79–92, 2022. doi MSRS
  47. Q. Ha, K. Watanabe, T. Karasawa, Y. Ushiku and T. Harada, “MFNet: Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes,” in Proc. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2017, pp. 5108–5115. doi
  48. X. Jia, C. Zhu, M. Li, W. Tang and W. Zhou, “LLVIP: A visible-infrared paired dataset for low-light vision,” in Proc. IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, 2021, pp. 3496–3504. CVF arXiv
  49. J. Cai, S. Gu and L. Zhang, “Learning a deep single image contrast enhancer from multi-exposure images,” IEEE Transactions on Image Processing, vol. 27, no. 4, pp. 2049–2062, 2018. doi
  50. X. Zhang, “Deep learning-based multi-focus image fusion: A survey and a comparative study,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 9, 2022. doi
  51. F. Xu, J. Liu, Y. Song, H. Sun and X. Wang, “Multi-exposure image fusion techniques: A comprehensive review,” Remote Sensing, vol. 14, no. 3, art. 771, 2022. doi
  52. Z. Zhao, H. Bai, J. Zhang, Y. Zhang, K. Zhang, S. Xu, D. Chen, R. Timofte and L. Van Gool, “Equivariant multi-modality image fusion,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 25912–25921. CVF arXiv
  53. L. Tang, Y. Wang, M. Gong, Z. Li, Y. Deng, X. Yi, C. Li, H. Xu, H. Zhang and J. Ma, “VideoFusion: A spatio-temporal collaborative network for multi-modal video fusion,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026, pp. 19559–19569. CVF
  54. N. Yokoya, C. Grohnfeldt and J. Chanussot, “Hyperspectral and multispectral data fusion: A comparative review of the recent literature,” IEEE Geoscience and Remote Sensing Magazine, 2017. doi
  55. H. R. Sheikh and A. C. Bovik, “Image information and visual quality,” IEEE Transactions on Image Processing, 2006. doi
  56. H. Xu et al., “U2Fusion,” official implementation accompanying the TPAMI paper, 2022. code
  57. K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” ICLR, 2015. arXiv
  58. J. Kirkpatrick et al., “Overcoming catastrophic forgetting in neural networks,” Proceedings of the National Academy of Sciences, 2017. doi
  59. Y. Zhang et al., “IFCNN,” official implementation and errata accompanying the Information Fusion paper, 2020. code
  60. K. He, X. Zhang, S. Ren and J. Sun, “Deep residual learning for image recognition,” CVPR, 2016. CVF
  61. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li and L. Fei-Fei, “ImageNet: A large-scale hierarchical image database,” CVPR, 2009. doi
  62. N. Silberman, D. Hoiem, P. Kohli and R. Fergus, “Indoor segmentation and support inference from RGBD images,” ECCV, 2012. project
  63. P. Dhariwal and A. Nichol, “Diffusion models beat GANs on image synthesis,” NeurIPS, 2021. arXiv
  64. J. Liu et al., “TarDAL,” official implementation accompanying the CVPR paper, 2022. code
  65. G. Jocher et al., “YOLOv5,” Ultralytics object-detection software, 2020 onward. code
  66. J. Ma et al., “FusionGAN,” official implementation accompanying the Information Fusion paper, 2019. code
  67. L. Tang, J. Yuan and J. Ma, “SeAFusion,” official implementation accompanying the Information Fusion paper, 2022. code
  68. T. Mertens, J. Kautz and F. Van Reeth, “Exposure Fusion,” in Proc. Pacific Graphics, 2007. paper · author project