U2Fusion: one unsupervised model across fusion tasks
Optional · paper guide 8 of 11 for DLCV 10 · Image Fusion: Complementary Images, Useful Outputs. Citation numbers refer to the reference list at the bottom of this page.
Navigate: this page and this track: Navigate: this page and this track
On this page
- Exposure fusion (Mertens, Kautz and Van Reeth)
- VIFB: a visible and infrared image fusion benchmark
- The infrared–visible fusion survey of Liu et al. (TPAMI 2025)
- DenseFuse: learn reconstruction, choose fusion in feature space
- DDFM: a diffusion prior with likelihood rectification
- FusionGAN: thermal intensity with an adversarial visible-image prior
- IFCNN: supervised fusion with a variable number of inputs
- U2Fusion: one unsupervised model across fusion tasks
- SwinFusion: attention within and across the sources
- SeAFusion: let semantic segmentation influence the fused image
- TarDAL: fuse for both appearance and object detection
Deep Learning for Computer Vision
Xu et al.’s TPAMI 2022 paper treats infrared–visible, medical, multi-exposure and multi-focus fusion as related information-preservation problems [32]. Its distinctive goal is one set of learned parameters, trained across tasks without ground-truth fused images. The following details refer to the journal version, which extends the earlier FusionDN design.
Challenge
Different fusion tasks appear to need incompatible rules. A warm infrared target should remain salient; an anatomical image should retain structure; an underexposed shot may contain useful content despite weak raw gradients; an out-of-focus region should contribute less than its sharp counterpart. A single hand-designed activity measure is unlikely to express all these preferences.
There are two separate obstacles. First, without fused targets, what should supervise the output? Second, when the same network learns another fusion task, how can it retain earlier capabilities? U2Fusion answers the first with an adaptively weighted similarity loss and the second with elastic weight consolidation, or EWC. “Unified” therefore means more than reusing an architecture with separately trained checkpoints.
Method

Figure source: [32], Public manuscript PDF, PDF page 4. Original PDF.
1. Generate an image, rather than a decision map. Concatenate the two single-channel inputs along the channel axis. For a patch of height and width , the input is and the output is . The fully convolutional generator uses dense feature connections, reflection padding and stride-one convolutions, without pooling. Dense connections make earlier features available to later layers and help information and gradients propagate.
Figure 8 and the official implementation specify an initial 44-channel convolution, five dense convolutions adding 44 channels each, and reconstruction convolutions reducing the resulting 264 channels to 128, 64, 32 and 1 [32], [56]. This gives ten convolutions. The prose’s reference to “first seven” dense layers is inconsistent with its diagram and the code; the explicit tensor path above follows the latter. Intermediate activations use a leaky rectifier with slope 0.2, and the final activation is tanh. Input/output scaling must match the implementation when reproducing it.
2. Measure source information using a fixed pretrained network. Each grayscale source is repeated into three channels and passed through VGG-16 [57]. Features are taken before its five max-pooling stages. For a patch, their spatial sizes are on each side, with channel counts . Shallow features respond to texture; deeper features provide larger-context structure. The feature extractor is used to define training supervision, not as a second generator.
The paper measures activity through a Laplacian on these feature maps:
is a source patch; is channel of the feature tensor at stage ; are that tensor’s height, width and channel count. is the spatial Laplacian and sums squared entries. Normalizing by the tensor size and averaging the stages gives a scalar activity score for each source patch. It is a designed feature statistic, not Shannon information or a learned estimate of physical sensor reliability.
3. Turn the measurements into preservation weights. For source patches , compute
is a task-specific scaling constant. The weights are nonnegative and sum to one. A smaller makes their difference more decisive. These are source weights for a training patch, not a dense per-pixel segmentation of which sensor is correct. Across many patches, they encourage the generator to preserve the source whose measured feature activity is larger.
4. Use complementary similarity terms. With generator output , define
denotes generator parameters, the expectation averages training pairs, indexes the two sources, SSIM compares structure and luminance statistics [16], MSE is mean squared pixel error, and controls their trade-off. The journal model uses [32]. SSIM encourages structural similarity while MSE constrains intensity differences. Neither term supplies a uniquely correct fused target, and a high weight does not guarantee every feature from that source survives.
Importantly, the journal objective does not retain FusionDN’s gradient-matching loss. The authors remove it to reduce false edges and add MSE to better constrain brightness. Do not implement U2Fusion by combining every loss mentioned in the historical discussion.
5. Learn tasks sequentially with EWC. Ordinary fine-tuning can overwrite useful parameters. EWC [58] penalizes movement of parameters estimated to matter to previous tasks:
is a parameter value after learning earlier tasks, its current value, an importance weight, and the consolidation strength. Parameters with larger are more strongly protected. The paper motivates using diagonal Fisher information and approximates it with an average squared gradient of the previous task’s loss at . For this general fusion loss, treat it as an importance approximation rather than an independently calibrated probabilistic uncertainty.
Importance is estimated before discarding the previous training data. The method does not need to replay all old training images during the next task, although data are used to estimate the protection weights. With several completed tasks, importance information must be accumulated according to the stated training procedure; retaining only the immediately preceding task’s unweighted parameters would not reproduce the intended consolidation.
The adopted order is multimodal → multi-exposure → multi-focus. Multimodal combines visible–infrared and PET–MRI examples as one training task. The journal setup crops patches, uses batch size 18, learning rate , 3/2/2 epochs for the three tasks, respectively, and [32]. These scales depend on the exact feature and intensity conventions; they are not portable defaults for arbitrary backbones.
6. Color and more than two inputs. Color images are converted to YCbCr, and the generator fuses luminance. Chroma is combined by a separate rule that favors stronger departures from neutral chroma. The paper writes, for either chroma channel,
are source chroma values in the paper’s 8-bit convention. A practical implementation must handle the zero denominator when both equal the neutral value, for example by returning . If only one source has color, its chroma supplies that part of the output. This is not end-to-end learned color fusion.
For a sequence, fuse two inputs, then fuse their intermediate result with the next input. This permits more than two sources but does not make the binary generator associative or order-independent. Errors or preferences can accumulate through the sequence.
What runs at inference. The learned fusion generator and the required color conversions run. VGG feature measurement, adaptive loss weights and EWC optimization are training mechanisms; they are not extra inference networks. This separation is central to understanding the model’s practical cost.
Results
The journal paper evaluates six collections across its tasks: TNO and RoadScene for visible–infrared, Harvard PET–MRI, the multi-exposure collection used in its training study and EMPA HDR, and Lytro multi-focus [32]. It introduces RoadScene with 221 aligned pairs. The quantitative visible–infrared comparisons use 20 TNO pairs and 45 RoadScene pairs; other images are shown qualitatively. Training, test selection and preprocessing therefore matter when comparing these tables with later papers.
Selected journal values illustrate both strengths and limits:
| Setting | U2Fusion result, as printed | Interpretation |
|---|---|---|
| TNO, Table 1 | CC ; SSIM ; PSNR | Source-comparison metrics under this paper’s definitions |
| RoadScene, Table 1 | CC ; SCD | It does not win SCD; FusionDN gives |
| Harvard, Table 2 | CC ; SSIM | Correlation/structural preservation, not diagnostic validation |
| Lytro, Table 4 | CC ; VIF | Strong results on a small multi-focus benchmark |
The SSIM values near 2 use a fusion-specific aggregation over sources. They are not ordinary single-reference SSIM values bounded above by 1. Likewise, the paper’s PSNR columns compare under its source-based evaluation; there is no hidden true fused image, and these large values should not be compared directly with restoration PSNR against clean ground truth. A reproduction must use the same toolkit and scaling.
The method also does not dominate every task metric: Table 3 gives 58.454 for its EMPA HDR PSNR column, while DeepFuse gives 58.518; Table 4 gives FusionDN a larger VIF than U2Fusion [32]. Report the full metric context instead of selecting only winning columns.
Ablations
EWC. Figures 19–20 compare sequential training with and without consolidation. Without EWC, learning later tasks changes earlier-task brightness and appearance; EWC alleviates this forgetting. The accompanying parameter-importance histograms describe the learned protection weights. These are qualitative and distributional analyses, not a table establishing a particular numerical AP-like gain for EWC.
Shared versus individual models. Figure 21 compares separately trained models with the unified model. The authors show examples where learning exposure handling helps overexposed regions in other tasks, and where multi-focus training helps sharp boundaries in exposure fusion. This supports cross-task transfer in those examples, not a guarantee that every added task improves all previous ones.
Adaptive weights. Figure 22 fixes both preservation weights to 0.5 and compares with the adaptive version. The equal-weight outputs lose details or retain overexposed regions, with blurred boundaries particularly visible in multi-focus examples. The paper provides images for this ablation; it does not provide a complete numerical table from which to invent a universal percentage improvement.
Training order. Table 5 supplies quantitative evidence. The adopted order gives Lytro CC 0.9723 / mean gradient 0.0677. Swapping the first two tasks gives 0.9724 / 0.0700. Starting with multi-focus and finishing with exposure gives 0.9803 / 0.0563 [32]. The last order has higher correlation but weaker mean gradient and visibly softer boundaries. Order still matters despite EWC, and the metrics disagree about what improved.
Journal changes from FusionDN. Figure 24 compares the earlier model with U2Fusion. Feature-based activity measurement replaces the earlier source-level information rule; the gradient loss is removed; MSE is added; and medical examples extend the first task. These changes are not all isolated numerically in a complete factorial study. The comparison supports the revised recipe, while limiting attribution to any single change.
Conclusion
U2Fusion unifies the training objective and the learned parameters across several fusion tasks. Fixed pretrained features define adaptive source preferences; structure and intensity losses train the generator; EWC reduces forgetting during sequential training. Its enduring contribution is to make “one model for several fusion tasks” a concrete optimization problem.
Problems
The feature activity statistic can reward noise or strong artifacts, and its source weights are scalar per training patch. A pretrained natural-image feature extractor is not guaranteed to measure medical or thermal importance correctly. The objective still expresses a designed preference, not a physical fused truth.
Registration is assumed, color is partly handled by a fixed rule, and sequential multi-input fusion is order-dependent. EWC reduces forgetting but does not eliminate it, as the training-order table shows. Small datasets and source-based metrics also limit claims of broad generalization. Later SwinFusion changes feature interaction, while task-aware approaches ask whether the fused image improves a specific downstream task; neither removes the need to check alignment, fidelity and evaluation assumptions.
References
- Y. Liu, X. Chen, Z. Wang, Z. J. Wang, R. K. Ward and X. Wang, “Deep learning for pixel-level image fusion: Recent advances and future prospects,” Information Fusion, vol. 42, pp. 158–173, 2018. doi
- J. Liu, G. Wu, Z. Liu, D. Wang, Z. Jiang, L. Ma, W. Zhong, X. Fan and R. Liu, “Infrared and visible image fusion: From data compatibility to task adaption,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 47, no. 4, pp. 2349–2369, 2025. doi arXiv
- J. Ma, Y. Ma and C. Li, “Infrared and visible image fusion methods and applications: A survey,” Information Fusion, vol. 45, pp. 153–178, 2019. doi
- P. E. Debevec and J. Malik, “Recovering high dynamic range radiance maps from photographs,” in Proc. SIGGRAPH, 1997, pp. 369–378. doi
- T. Mertens, J. Kautz and F. Van Reeth, “Exposure fusion: A simple and practical alternative to high dynamic range photography,” Computer Graphics Forum, vol. 28, no. 1, pp. 161–171, 2009. doi
- S. Xu, X. Wei, C. Zhang, J. Liu and J. Zhang, “MFFW: A new dataset for multi-focus image fusion,” arXiv:2002.04780, 2020. arXiv
- X. Zhang, “Multi-focus image fusion: A benchmark,” arXiv:2005.01116, 2020. arXiv
- P. J. Burt and E. H. Adelson, “The Laplacian pyramid as a compact image code,” IEEE Transactions on Communications, vol. 31, no. 4, pp. 532–540, 1983. doi
- P. J. Burt and R. J. Kolczynski, “Enhanced image capture through fusion,” in Proc. 4th International Conference on Computer Vision (ICCV), 1993, pp. 173–182. doi
- P. J. Burt and E. H. Adelson, “A multiresolution spline with application to image mosaics,” ACM Transactions on Graphics, vol. 2, no. 4, pp. 217–236, 1983. doi
- H. Li, B. S. Manjunath and S. K. Mitra, “Multisensor image fusion using the wavelet transform,” Graphical Models and Image Processing, vol. 57, no. 3, pp. 235–245, 1995. doi
- S. Li, X. Kang and J. Hu, “Image fusion with guided filtering,” IEEE Transactions on Image Processing, vol. 22, no. 7, pp. 2864–2875, 2013. doi
- K. He, J. Sun and X. Tang, “Guided image filtering,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, no. 6, pp. 1397–1409, 2013. doi
- B. Yang and S. Li, “Multifocus image fusion and restoration with sparse representation,” IEEE Transactions on Instrumentation and Measurement, vol. 59, no. 4, pp. 884–892, 2010. doi
- M. Nejati, S. Samavi and S. Shirani, “Multi-focus image fusion using dictionary-based sparse representation,” Information Fusion, vol. 25, pp. 72–84, 2015. doi
- Z. Wang, A. C. Bovik, H. R. Sheikh and E. P. Simoncelli, “Image quality assessment: From error visibility to structural similarity,” IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600–612, 2004. doi
- X. Zhang, P. Ye and G. Xiao, “VIFB: A visible and infrared image fusion benchmark,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2020. CVF arXiv
- X. Zhang, “Benchmarking and comparing multi-exposure image fusion algorithms,” Information Fusion, vol. 74, pp. 111–131, 2021. doi arXiv
- A. M. Eskicioglu and P. S. Fisher, “Image quality measures and their performance,” IEEE Transactions on Communications, vol. 43, no. 12, pp. 2959–2965, 1995. doi
- G. Qu, D. Zhang and P. Yan, “Information measure for performance of image fusion,” Electronics Letters, vol. 38, no. 7, pp. 313–315, 2002. doi
- M. Hossny, S. Nahavandi and D. Creighton, “Comments on ‘Information measure for performance of image fusion’,” Electronics Letters, vol. 44, no. 18, pp. 1066–1067, 2008. doi
- C. S. Xydeas and V. Petrović, “Objective image fusion performance measure,” Electronics Letters, vol. 36, no. 4, pp. 308–309, 2000. doi
- G. Piella and H. Heijmans, “A new quality metric for image fusion,” in Proc. IEEE International Conference on Image Processing (ICIP), 2003, vol. 3, pp. III-173–176. doi
- Y. Han, Y. Cai, Y. Cao and X. Xu, “A new image fusion performance metric based on visual information fidelity,” Information Fusion, vol. 14, no. 2, pp. 127–135, 2013. doi
- K. Ma, K. Zeng and Z. Wang, “Perceptual quality assessment for multi-exposure image fusion,” IEEE Transactions on Image Processing, vol. 24, no. 11, pp. 3345–3356, 2015. doi
- Z. Liu, E. Blasch, Z. Xue, J. Zhao, R. Laganiere and W. Wu, “Objective assessment of multiresolution image fusion algorithms for context enhancement in night vision: A comparative study,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 34, no. 1, pp. 94–109, 2012. doi
- J. Liu, X. Fan, Z. Huang, G. Wu, R. Liu, W. Zhong and Z. Luo, “Target-aware dual adversarial learning and a multi-scenario multi-modality benchmark to fuse infrared and visible for object detection,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 5802–5811. CVF arXiv M3FD
- J. Liu, Z. Liu, G. Wu, L. Ma, R. Liu, W. Zhong, Z. Luo and X. Fan, “Multi-interactive feature learning and a full-time multi-modality benchmark for image fusion and segmentation,” in Proc. IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 8115–8124. CVF arXiv
- Y. Zhang, Y. Liu, P. Sun, H. Yan, X. Zhao and L. Zhang, “IFCNN: A general image fusion framework based on convolutional neural network,” Information Fusion, vol. 54, pp. 99–118, 2020. doi
- H. Li and X.-J. Wu, “DenseFuse: A fusion approach to infrared and visible images,” IEEE Transactions on Image Processing, vol. 28, no. 5, pp. 2614–2623, 2019. doi arXiv
- K. R. Prabhakar, V. S. Srikar and R. V. Babu, “DeepFuse: A deep unsupervised approach for exposure fusion with extreme exposure image pairs,” in Proc. IEEE International Conference on Computer Vision (ICCV), 2017, pp. 4724–4732. doi CVF
- H. Xu, J. Ma, J. Jiang, X. Guo and H. Ling, “U2Fusion: A unified unsupervised image fusion network,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 1, pp. 502–518, 2022. doi RoadScene
- L. Tang, J. Yuan and J. Ma, “Image fusion in the loop of high-level vision tasks: A semantic-aware real-time infrared and visible image fusion network,” Information Fusion, vol. 82, pp. 28–42, 2022. doi
- J. Ma, L. Tang, F. Fan, J. Huang, X. Mei and Y. Ma, “SwinFusion: Cross-domain long-range learning for general image fusion via Swin Transformer,” IEEE/CAA Journal of Automatica Sinica, vol. 9, no. 7, pp. 1200–1217, 2022. doi
- X. Zhang and Y. Demiris, “Visible and infrared image fusion using deep learning,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 8, pp. 10535–10554, 2023. doi
- Y. Liu, X. Chen, H. Peng and Z. Wang, “Multi-focus image fusion with a deep convolutional neural network,” Information Fusion, vol. 36, pp. 191–207, 2017. doi
- K. Ma, Z. Duanmu, H. Zhu, Y. Fang and Z. Wang, “Deep guided learning for fast multi-exposure image fusion,” IEEE Transactions on Image Processing, vol. 29, pp. 2808–2819, 2020. doi
- Z. Zhao, H. Bai, J. Zhang, Y. Zhang, S. Xu, Z. Lin, R. Timofte and L. Van Gool, “CDDFuse: Correlation-driven dual-branch feature decomposition for multi-modality image fusion,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 5906–5916. CVF arXiv
- J. Ma, W. Yu, P. Liang, C. Li and J. Jiang, “FusionGAN: A generative adversarial network for infrared and visible image fusion,” Information Fusion, vol. 48, pp. 11–26, 2019. doi · public article transcription
- H. Bai, J. Zhang, Z. Zhao, Y. Wu, L. Deng, Y. Cui, T. Feng and S. Xu, “Task-driven image fusion with learnable fusion loss,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025, pp. 7457–7468. CVF arXiv
- Z. Zhao, H. Bai, Y. Zhu, J. Zhang, S. Xu, Y. Zhang, K. Zhang, D. Meng, R. Timofte and L. Van Gool, “DDFM: Denoising diffusion model for multi-modality image fusion,” in Proc. IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 8082–8093. CVF arXiv
- X. Yi, H. Xu, H. Zhang, L. Tang and J. Ma, “Text-IF: Leveraging semantic text guidance for degradation-aware and interactive image fusion,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 27026–27035. CVF arXiv
- D. Wang, J. Liu, X. Fan and R. Liu, “Unsupervised misaligned infrared and visible image fusion via cross-modality image generation and registration,” in Proc. International Joint Conference on Artificial Intelligence (IJCAI), 2022, pp. 3508–3515. doi arXiv
- L. Tang, Y. Deng, Y. Ma, J. Huang and J. Ma, “SuperFusion: A versatile image registration and fusion network with semantic awareness,” IEEE/CAA Journal of Automatica Sinica, vol. 9, no. 12, pp. 2121–2137, 2022. doi
- A. Toet, “The TNO multiband image data collection,” Data in Brief, vol. 15, pp. 249–251, 2017. doi
- L. Tang, J. Yuan, H. Zhang, X. Jiang and J. Ma, “PIAFusion: A progressive infrared and visible image fusion network based on illumination aware,” Information Fusion, vol. 83–84, pp. 79–92, 2022. doi MSRS
- Q. Ha, K. Watanabe, T. Karasawa, Y. Ushiku and T. Harada, “MFNet: Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes,” in Proc. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2017, pp. 5108–5115. doi
- X. Jia, C. Zhu, M. Li, W. Tang and W. Zhou, “LLVIP: A visible-infrared paired dataset for low-light vision,” in Proc. IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, 2021, pp. 3496–3504. CVF arXiv
- J. Cai, S. Gu and L. Zhang, “Learning a deep single image contrast enhancer from multi-exposure images,” IEEE Transactions on Image Processing, vol. 27, no. 4, pp. 2049–2062, 2018. doi
- X. Zhang, “Deep learning-based multi-focus image fusion: A survey and a comparative study,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 9, 2022. doi
- F. Xu, J. Liu, Y. Song, H. Sun and X. Wang, “Multi-exposure image fusion techniques: A comprehensive review,” Remote Sensing, vol. 14, no. 3, art. 771, 2022. doi
- Z. Zhao, H. Bai, J. Zhang, Y. Zhang, K. Zhang, S. Xu, D. Chen, R. Timofte and L. Van Gool, “Equivariant multi-modality image fusion,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 25912–25921. CVF arXiv
- L. Tang, Y. Wang, M. Gong, Z. Li, Y. Deng, X. Yi, C. Li, H. Xu, H. Zhang and J. Ma, “VideoFusion: A spatio-temporal collaborative network for multi-modal video fusion,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026, pp. 19559–19569. CVF
- N. Yokoya, C. Grohnfeldt and J. Chanussot, “Hyperspectral and multispectral data fusion: A comparative review of the recent literature,” IEEE Geoscience and Remote Sensing Magazine, 2017. doi
- H. R. Sheikh and A. C. Bovik, “Image information and visual quality,” IEEE Transactions on Image Processing, 2006. doi
- H. Xu et al., “U2Fusion,” official implementation accompanying the TPAMI paper, 2022. code
- K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” ICLR, 2015. arXiv
- J. Kirkpatrick et al., “Overcoming catastrophic forgetting in neural networks,” Proceedings of the National Academy of Sciences, 2017. doi
- Y. Zhang et al., “IFCNN,” official implementation and errata accompanying the Information Fusion paper, 2020. code
- K. He, X. Zhang, S. Ren and J. Sun, “Deep residual learning for image recognition,” CVPR, 2016. CVF
- J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li and L. Fei-Fei, “ImageNet: A large-scale hierarchical image database,” CVPR, 2009. doi
- N. Silberman, D. Hoiem, P. Kohli and R. Fergus, “Indoor segmentation and support inference from RGBD images,” ECCV, 2012. project
- P. Dhariwal and A. Nichol, “Diffusion models beat GANs on image synthesis,” NeurIPS, 2021. arXiv
- J. Liu et al., “TarDAL,” official implementation accompanying the CVPR paper, 2022. code
- G. Jocher et al., “YOLOv5,” Ultralytics object-detection software, 2020 onward. code
- J. Ma et al., “FusionGAN,” official implementation accompanying the Information Fusion paper, 2019. code
- L. Tang, J. Yuan and J. Ma, “SeAFusion,” official implementation accompanying the Information Fusion paper, 2022. code
- T. Mertens, J. Kautz and F. Van Reeth, “Exposure Fusion,” in Proc. Pacific Graphics, 2007. paper · author project