IFCNN: supervised fusion with a variable number of inputs
Optional · paper guide 7 of 11 for DLCV 10 · Image Fusion: Complementary Images, Useful Outputs. Citation numbers refer to the reference list at the bottom of this page.
Navigate: this page and this track: Navigate: this page and this track
On this page
- Exposure fusion (Mertens, Kautz and Van Reeth)
- VIFB: a visible and infrared image fusion benchmark
- The infrared–visible fusion survey of Liu et al. (TPAMI 2025)
- DenseFuse: learn reconstruction, choose fusion in feature space
- DDFM: a diffusion prior with likelihood rectification
- FusionGAN: thermal intensity with an adversarial visible-image prior
- IFCNN: supervised fusion with a variable number of inputs
- U2Fusion: one unsupervised model across fusion tasks
- SwinFusion: attention within and across the sources
- SeAFusion: let semantic segmentation influence the fused image
- TarDAL: fuse for both appearance and object detection
Deep Learning for Computer Vision
Zhang et al.’s Information Fusion 2020 paper learns a compact fusion network from synthetic multi-focus examples, then evaluates it across several fusion scenarios [29]. Its distinguishing choices are a transferred first convolution, a parameter-free feature merger and a reconstruction-plus-perceptual training objective.
Challenge
A model that concatenates two encoded images and learns a fixed-width fusion convolution is tied to that input count. IFCNN instead wants a feature operation that can combine several sources without changing the learned parameter shapes. It also wants training targets despite the absence of correct fused images for most real multimodal pairs.
The solution has two parts: use element-wise feature operations for variable input count, and generate supervised multi-focus training pairs from sharp RGB images and depth maps. Generalization to other fusion types is then tested rather than directly supervised with their true fused outputs.
Method

Figure source: [29], Official implementation figure, commit ee21b644acca914bbb326daab115c458a67a773a. Original source.
Transferred feature extraction. Each source passes through the same feature extractor. CONV1 uses the first convolution of a ResNet101 pretrained on ImageNet [60], [61]. It maps three input channels to 64 feature channels with a kernel. Its weights remain fixed during the described training. The stride is changed to one and padding to three, preserving spatial size instead of retaining the classification network’s downsampling.
CONV2 applies a learned convolution with 64 output channels, batch normalization and a rectifier. It adapts the transferred low-level features to fusion. For an source, the feature tensor is . All sources use the same parameters, rather than one independently learned encoder per input index.
Feature merger. For source features of identical shape, the paper studies element-wise maximum, sum and mean:
is the source count; the maximum is taken independently at each position and channel. These operations add no learned fusion parameters and keep the output shape fixed as changes. Maximum favors strong responses; mean controls feature scale but blends responses; sum changes its magnitude with input count. None of these operations estimates alignment.
Reconstruction. CONV3 is another 64-channel adaptation layer with normalization and activation. CONV4 is a convolution mapping the fused features to three image channels. Its stride is one and padding zero, as clarified in the official errata; the paper’s printed zero stride is a typo [59]. No output normalization or activation is appended in the described architecture.
Grayscale images can be repeated into three channels for the shared RGB path. Color handling and postprocessing must follow the chosen evaluation setup. Accepting several input tensors is also different from proving equal quality for arbitrary long source sequences.
Synthetic supervision. The training data come from RGB/depth pairs in NYU Depth V2 [62]. Blur a sharp RGB image , threshold its depth map to make a focus mask , and construct complementary sources:
is a Gaussian-blurred image and means element-wise multiplication. The original sharp image supplies the target. Random depth thresholds, crops and flips create varied examples; later training also changes color and sometimes converts to grayscale. This is supervised learning, even though no human drew the fused targets.
Loss and training stages. The initial objective is mean squared RGB reconstruction error. Fine-training adds a perceptual loss from the last convolutional features of a fixed pretrained ResNet101:
is the generated image, the sharp target, the fixed perceptual feature extractor, and its channel count and spatial dimensions. The equal coefficients follow the reported experiment [29]; different feature normalization would change their relative importance. A perceptual target encourages learned feature similarity, not a guarantee of better fidelity on every metric.
The paper pretrains with pixel loss for 5,000 iterations at batch 64, then freezes batch-normalization parameters and fine-trains with both losses for 60,000 iterations at batch 32 [29]. The first transferred convolution remains fixed. Training and inference roles should therefore be kept separate: the deep ResNet used for the perceptual loss is absent from the deployed fusion path.
Results
The study evaluates multi-focus, infrared–visible, medical and multi-exposure collections [29]. On the multi-focus set, Table 1 gives IFCNN-MAX VIFF 0.9823, ISSIM 0.6293 and SF 19.42. On the infrared–visible set, Table 2 gives VIFF 0.6405 and SF 11.31. These use the paper’s metric conventions and scales; they are not equivalent to single-reference SSIM or downstream accuracy.
For multi-exposure fusion, the preferred variant changes. Table 4 gives IFCNN-MEAN MESSIM 0.8786, compared with 0.8151 for IFCNN-MAX, while GFF gives 0.9204 [29]. A single merging operation does not dominate every task.
Table 5 reports about 0.02 seconds for a pair on the stated Intel i7-3770K / NVIDIA TITAN X platform [29]. Competing rows use different input dimensions or execution paths, so retain those details rather than turning the table into a universal speed ranking.
Ablations
The BASELINE-MAX variant uses the pixel-loss training baseline; IFCNN-MAX adds perceptual training. In Table 1, VIFF changes from 0.9809 to 0.9823, NMI from 0.8664 to 0.9034, and SF from 19.33 to 19.42, while ISSIM falls from 0.6318 to 0.6293 [29]. The feature loss improves several criteria without improving all of them.
This trade-off is sharper on infrared–visible data: Table 2 gives the baseline VIFF 0.6621 / ISSIM 0.5002, compared with 0.6405 / 0.4956 for IFCNN-MAX, even though its SF rises from 11.08 to 11.31 [29]. The paper’s generalization and visual claims should be read alongside those declines.
Comparisons of MAX, SUM and MEAN additionally test the merger. On multi-exposure data, mean fusion gives a stronger structural score than max fusion, which gives much higher spatial frequency. That difference supports the chapter’s lesson: selecting detail and blending exposure are different preferences.
Conclusion
IFCNN provides a shared convolutional feature path with a merger whose parameter shapes do not depend on source count. Synthetic supervised training makes learning possible, while the evaluated task-specific merging choices determine how the learned features are combined.
Problems
The synthetic blur/mask process does not reproduce every real defocus boundary, sensor response or misalignment. Its perceptual features come from natural-image classification and can favor some structures over others. Several metrics decline when the perceptual loss is added, so “perceptual” should not be read as a universal improvement.
Finally, accepting different modalities through one architecture does not mean that one merger or one benchmark validates every application. Check input conventions, source count, registration and evaluation independently, and retain the official corrections when reproducing the network.
References
- Y. Liu, X. Chen, Z. Wang, Z. J. Wang, R. K. Ward and X. Wang, “Deep learning for pixel-level image fusion: Recent advances and future prospects,” Information Fusion, vol. 42, pp. 158–173, 2018. doi
- J. Liu, G. Wu, Z. Liu, D. Wang, Z. Jiang, L. Ma, W. Zhong, X. Fan and R. Liu, “Infrared and visible image fusion: From data compatibility to task adaption,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 47, no. 4, pp. 2349–2369, 2025. doi arXiv
- J. Ma, Y. Ma and C. Li, “Infrared and visible image fusion methods and applications: A survey,” Information Fusion, vol. 45, pp. 153–178, 2019. doi
- P. E. Debevec and J. Malik, “Recovering high dynamic range radiance maps from photographs,” in Proc. SIGGRAPH, 1997, pp. 369–378. doi
- T. Mertens, J. Kautz and F. Van Reeth, “Exposure fusion: A simple and practical alternative to high dynamic range photography,” Computer Graphics Forum, vol. 28, no. 1, pp. 161–171, 2009. doi
- S. Xu, X. Wei, C. Zhang, J. Liu and J. Zhang, “MFFW: A new dataset for multi-focus image fusion,” arXiv:2002.04780, 2020. arXiv
- X. Zhang, “Multi-focus image fusion: A benchmark,” arXiv:2005.01116, 2020. arXiv
- P. J. Burt and E. H. Adelson, “The Laplacian pyramid as a compact image code,” IEEE Transactions on Communications, vol. 31, no. 4, pp. 532–540, 1983. doi
- P. J. Burt and R. J. Kolczynski, “Enhanced image capture through fusion,” in Proc. 4th International Conference on Computer Vision (ICCV), 1993, pp. 173–182. doi
- P. J. Burt and E. H. Adelson, “A multiresolution spline with application to image mosaics,” ACM Transactions on Graphics, vol. 2, no. 4, pp. 217–236, 1983. doi
- H. Li, B. S. Manjunath and S. K. Mitra, “Multisensor image fusion using the wavelet transform,” Graphical Models and Image Processing, vol. 57, no. 3, pp. 235–245, 1995. doi
- S. Li, X. Kang and J. Hu, “Image fusion with guided filtering,” IEEE Transactions on Image Processing, vol. 22, no. 7, pp. 2864–2875, 2013. doi
- K. He, J. Sun and X. Tang, “Guided image filtering,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, no. 6, pp. 1397–1409, 2013. doi
- B. Yang and S. Li, “Multifocus image fusion and restoration with sparse representation,” IEEE Transactions on Instrumentation and Measurement, vol. 59, no. 4, pp. 884–892, 2010. doi
- M. Nejati, S. Samavi and S. Shirani, “Multi-focus image fusion using dictionary-based sparse representation,” Information Fusion, vol. 25, pp. 72–84, 2015. doi
- Z. Wang, A. C. Bovik, H. R. Sheikh and E. P. Simoncelli, “Image quality assessment: From error visibility to structural similarity,” IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600–612, 2004. doi
- X. Zhang, P. Ye and G. Xiao, “VIFB: A visible and infrared image fusion benchmark,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2020. CVF arXiv
- X. Zhang, “Benchmarking and comparing multi-exposure image fusion algorithms,” Information Fusion, vol. 74, pp. 111–131, 2021. doi arXiv
- A. M. Eskicioglu and P. S. Fisher, “Image quality measures and their performance,” IEEE Transactions on Communications, vol. 43, no. 12, pp. 2959–2965, 1995. doi
- G. Qu, D. Zhang and P. Yan, “Information measure for performance of image fusion,” Electronics Letters, vol. 38, no. 7, pp. 313–315, 2002. doi
- M. Hossny, S. Nahavandi and D. Creighton, “Comments on ‘Information measure for performance of image fusion’,” Electronics Letters, vol. 44, no. 18, pp. 1066–1067, 2008. doi
- C. S. Xydeas and V. Petrović, “Objective image fusion performance measure,” Electronics Letters, vol. 36, no. 4, pp. 308–309, 2000. doi
- G. Piella and H. Heijmans, “A new quality metric for image fusion,” in Proc. IEEE International Conference on Image Processing (ICIP), 2003, vol. 3, pp. III-173–176. doi
- Y. Han, Y. Cai, Y. Cao and X. Xu, “A new image fusion performance metric based on visual information fidelity,” Information Fusion, vol. 14, no. 2, pp. 127–135, 2013. doi
- K. Ma, K. Zeng and Z. Wang, “Perceptual quality assessment for multi-exposure image fusion,” IEEE Transactions on Image Processing, vol. 24, no. 11, pp. 3345–3356, 2015. doi
- Z. Liu, E. Blasch, Z. Xue, J. Zhao, R. Laganiere and W. Wu, “Objective assessment of multiresolution image fusion algorithms for context enhancement in night vision: A comparative study,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 34, no. 1, pp. 94–109, 2012. doi
- J. Liu, X. Fan, Z. Huang, G. Wu, R. Liu, W. Zhong and Z. Luo, “Target-aware dual adversarial learning and a multi-scenario multi-modality benchmark to fuse infrared and visible for object detection,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 5802–5811. CVF arXiv M3FD
- J. Liu, Z. Liu, G. Wu, L. Ma, R. Liu, W. Zhong, Z. Luo and X. Fan, “Multi-interactive feature learning and a full-time multi-modality benchmark for image fusion and segmentation,” in Proc. IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 8115–8124. CVF arXiv
- Y. Zhang, Y. Liu, P. Sun, H. Yan, X. Zhao and L. Zhang, “IFCNN: A general image fusion framework based on convolutional neural network,” Information Fusion, vol. 54, pp. 99–118, 2020. doi
- H. Li and X.-J. Wu, “DenseFuse: A fusion approach to infrared and visible images,” IEEE Transactions on Image Processing, vol. 28, no. 5, pp. 2614–2623, 2019. doi arXiv
- K. R. Prabhakar, V. S. Srikar and R. V. Babu, “DeepFuse: A deep unsupervised approach for exposure fusion with extreme exposure image pairs,” in Proc. IEEE International Conference on Computer Vision (ICCV), 2017, pp. 4724–4732. doi CVF
- H. Xu, J. Ma, J. Jiang, X. Guo and H. Ling, “U2Fusion: A unified unsupervised image fusion network,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 1, pp. 502–518, 2022. doi RoadScene
- L. Tang, J. Yuan and J. Ma, “Image fusion in the loop of high-level vision tasks: A semantic-aware real-time infrared and visible image fusion network,” Information Fusion, vol. 82, pp. 28–42, 2022. doi
- J. Ma, L. Tang, F. Fan, J. Huang, X. Mei and Y. Ma, “SwinFusion: Cross-domain long-range learning for general image fusion via Swin Transformer,” IEEE/CAA Journal of Automatica Sinica, vol. 9, no. 7, pp. 1200–1217, 2022. doi
- X. Zhang and Y. Demiris, “Visible and infrared image fusion using deep learning,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 8, pp. 10535–10554, 2023. doi
- Y. Liu, X. Chen, H. Peng and Z. Wang, “Multi-focus image fusion with a deep convolutional neural network,” Information Fusion, vol. 36, pp. 191–207, 2017. doi
- K. Ma, Z. Duanmu, H. Zhu, Y. Fang and Z. Wang, “Deep guided learning for fast multi-exposure image fusion,” IEEE Transactions on Image Processing, vol. 29, pp. 2808–2819, 2020. doi
- Z. Zhao, H. Bai, J. Zhang, Y. Zhang, S. Xu, Z. Lin, R. Timofte and L. Van Gool, “CDDFuse: Correlation-driven dual-branch feature decomposition for multi-modality image fusion,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 5906–5916. CVF arXiv
- J. Ma, W. Yu, P. Liang, C. Li and J. Jiang, “FusionGAN: A generative adversarial network for infrared and visible image fusion,” Information Fusion, vol. 48, pp. 11–26, 2019. doi · public article transcription
- H. Bai, J. Zhang, Z. Zhao, Y. Wu, L. Deng, Y. Cui, T. Feng and S. Xu, “Task-driven image fusion with learnable fusion loss,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025, pp. 7457–7468. CVF arXiv
- Z. Zhao, H. Bai, Y. Zhu, J. Zhang, S. Xu, Y. Zhang, K. Zhang, D. Meng, R. Timofte and L. Van Gool, “DDFM: Denoising diffusion model for multi-modality image fusion,” in Proc. IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 8082–8093. CVF arXiv
- X. Yi, H. Xu, H. Zhang, L. Tang and J. Ma, “Text-IF: Leveraging semantic text guidance for degradation-aware and interactive image fusion,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 27026–27035. CVF arXiv
- D. Wang, J. Liu, X. Fan and R. Liu, “Unsupervised misaligned infrared and visible image fusion via cross-modality image generation and registration,” in Proc. International Joint Conference on Artificial Intelligence (IJCAI), 2022, pp. 3508–3515. doi arXiv
- L. Tang, Y. Deng, Y. Ma, J. Huang and J. Ma, “SuperFusion: A versatile image registration and fusion network with semantic awareness,” IEEE/CAA Journal of Automatica Sinica, vol. 9, no. 12, pp. 2121–2137, 2022. doi
- A. Toet, “The TNO multiband image data collection,” Data in Brief, vol. 15, pp. 249–251, 2017. doi
- L. Tang, J. Yuan, H. Zhang, X. Jiang and J. Ma, “PIAFusion: A progressive infrared and visible image fusion network based on illumination aware,” Information Fusion, vol. 83–84, pp. 79–92, 2022. doi MSRS
- Q. Ha, K. Watanabe, T. Karasawa, Y. Ushiku and T. Harada, “MFNet: Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes,” in Proc. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2017, pp. 5108–5115. doi
- X. Jia, C. Zhu, M. Li, W. Tang and W. Zhou, “LLVIP: A visible-infrared paired dataset for low-light vision,” in Proc. IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, 2021, pp. 3496–3504. CVF arXiv
- J. Cai, S. Gu and L. Zhang, “Learning a deep single image contrast enhancer from multi-exposure images,” IEEE Transactions on Image Processing, vol. 27, no. 4, pp. 2049–2062, 2018. doi
- X. Zhang, “Deep learning-based multi-focus image fusion: A survey and a comparative study,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 9, 2022. doi
- F. Xu, J. Liu, Y. Song, H. Sun and X. Wang, “Multi-exposure image fusion techniques: A comprehensive review,” Remote Sensing, vol. 14, no. 3, art. 771, 2022. doi
- Z. Zhao, H. Bai, J. Zhang, Y. Zhang, K. Zhang, S. Xu, D. Chen, R. Timofte and L. Van Gool, “Equivariant multi-modality image fusion,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 25912–25921. CVF arXiv
- L. Tang, Y. Wang, M. Gong, Z. Li, Y. Deng, X. Yi, C. Li, H. Xu, H. Zhang and J. Ma, “VideoFusion: A spatio-temporal collaborative network for multi-modal video fusion,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026, pp. 19559–19569. CVF
- N. Yokoya, C. Grohnfeldt and J. Chanussot, “Hyperspectral and multispectral data fusion: A comparative review of the recent literature,” IEEE Geoscience and Remote Sensing Magazine, 2017. doi
- H. R. Sheikh and A. C. Bovik, “Image information and visual quality,” IEEE Transactions on Image Processing, 2006. doi
- H. Xu et al., “U2Fusion,” official implementation accompanying the TPAMI paper, 2022. code
- K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” ICLR, 2015. arXiv
- J. Kirkpatrick et al., “Overcoming catastrophic forgetting in neural networks,” Proceedings of the National Academy of Sciences, 2017. doi
- Y. Zhang et al., “IFCNN,” official implementation and errata accompanying the Information Fusion paper, 2020. code
- K. He, X. Zhang, S. Ren and J. Sun, “Deep residual learning for image recognition,” CVPR, 2016. CVF
- J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li and L. Fei-Fei, “ImageNet: A large-scale hierarchical image database,” CVPR, 2009. doi
- N. Silberman, D. Hoiem, P. Kohli and R. Fergus, “Indoor segmentation and support inference from RGBD images,” ECCV, 2012. project
- P. Dhariwal and A. Nichol, “Diffusion models beat GANs on image synthesis,” NeurIPS, 2021. arXiv
- J. Liu et al., “TarDAL,” official implementation accompanying the CVPR paper, 2022. code
- G. Jocher et al., “YOLOv5,” Ultralytics object-detection software, 2020 onward. code
- J. Ma et al., “FusionGAN,” official implementation accompanying the Information Fusion paper, 2019. code
- L. Tang, J. Yuan and J. Ma, “SeAFusion,” official implementation accompanying the Information Fusion paper, 2022. code
- T. Mertens, J. Kautz and F. Van Reeth, “Exposure Fusion,” in Proc. Pacific Graphics, 2007. paper · author project