Full search page →

SeAFusion: let semantic segmentation influence the fused image

Optional · paper guide 10 of 11 for DLCV 10 · Image Fusion: Complementary Images, Useful Outputs. Citation numbers refer to the reference list at the bottom of this page.

Navigate: this page and this track: Navigate: this page and this track

On this page

Supplementary notes · 6

  1. The defocus spread effect
  2. Building and collapsing a Laplacian pyramid
  3. Sparse-representation fusion
  4. Mutual information as a fusion metric
  5. Training signals of representative fusion networks
  6. VIF and source-referenced fusion scores

Paper guides · 11

  1. Exposure fusion (Mertens, Kautz and Van Reeth)
  2. VIFB: a visible and infrared image fusion benchmark
  3. The infrared–visible fusion survey of Liu et al. (TPAMI 2025)
  4. DenseFuse: learn reconstruction, choose fusion in feature space
  5. DDFM: a diffusion prior with likelihood rectification
  6. FusionGAN: thermal intensity with an adversarial visible-image prior
  7. IFCNN: supervised fusion with a variable number of inputs
  8. U2Fusion: one unsupervised model across fusion tasks
  9. SwinFusion: attention within and across the sources
  10. SeAFusion: let semantic segmentation influence the fused image
  11. TarDAL: fuse for both appearance and object detection

Deep Learning for Computer Vision

  1. 02 · Deep Image Classification
View the whole course tree →

Challenge

An image can look sharp and score well on source-based metrics while still hiding a pedestrian from a segmentation system. SeAFusion [33] places fusion inside a high-level learning loop. It retains source intensity and gradients, then uses a semantic segmentation network to supply a task-dependent training signal. The fused image remains a single image that can be inspected and supplied to other systems; it is not merely a feature tensor inside one detector.

This introduction distinguishes the journal’s problem statement from the precise settings of the opened official PyTorch implementation [67]. The latter is executable evidence about a particular release, rather than permission to invent unavailable original table values.

Method

The fused image feeds segmentation, while semantic loss returns a training signal to fusion; segmentation predictions and fusion supervision have distinct roles.
Official project architecture diagram — The fused image feeds segmentation, while semantic loss returns a training signal to fusion; segmentation predictions and fusion supervision have distinct roles.

Figure source: [33], Official implementation figure, commit cca928453e6969b0c37ea9fbe172ee51aa63c8b8. Original source.

Two branches preserve the modality distinction. The visible RGB image is converted to luminance and chrominance; fusion operates on visible luminance and one infrared channel. Each branch starts with a 3 × 3 convolution producing 16 channels, then has two gradient residual dense blocks producing 32 and 48 channels. For input batch size BB and dimensions H,WH,W, the two final maps are each B×48×H×WB\times48\times H\times W. Stride one and padding preserve alignment.

Inside a block, a dense convolution branch concatenates its intermediate features, then uses a 1 × 1 projection to control channel count. A parallel branch applies horizontal and vertical Sobel-shaped filters, sums their absolute responses and projects them with another 1 × 1 convolution. The two branches are added and passed through LeakyReLU. The dense path learns contextual combinations; the gradient path emphasizes boundaries that a small real-time network might otherwise smooth away.

There is an implementation subtlety: the model’s Sobel-shaped convolution weights are initialized from the filters but are ordinary learned parameters in the opened release. In contrast, the Sobel kernels inside the loss are explicitly fixed. “Gradient guidance” does not mean that every filter carrying that name remains immutable during training.

The two 48-channel maps concatenate to 96 channels. Four decoder convolutions reduce the width through 64, 32 and 16 to one output channel. The final transformed tanh produces luminance in [0,1][0,1]. Classes named ConvBnLeakyRelu2d and ConvBnTanh2d instantiate batch-normalization modules, but their actual forward methods in this release do not call them. A reproduction should follow executed operations rather than infer behavior from class names. The fused luminance is combined with the original visible chrominance to form the color image.

A source-preservation objective supplies the base signal. In the released code,

g(X)=∣Kx∗X∣+∣Ky∗X∣,Li=MAE⁡(F,max⁡(Y,I)),Lg=MAE⁡(g(F),max⁡(g(Y),g(I))),Lfusion=Li+10Lg.\begin{aligned} g(X)&=|K_x*X|+|K_y*X|,\\ \mathcal L_i&=\operatorname{MAE}(F,\max(Y,I)),\\ \mathcal L_g&=\operatorname{MAE}(g(F),\max(g(Y),g(I))),\\ \mathcal L_{\rm fusion}&=\mathcal L_i+10\mathcal L_g. \end{aligned}

Here YY is visible luminance, II is infrared intensity, FF is fused luminance, Kx,KyK_x,K_y are fixed Sobel kernels, ∗* denotes convolution and MAE averages absolute residuals. Maximum and absolute values are elementwise. The intensity target protects bright evidence from either source. The gradient target protects strong edges from either source. This is a deliberately chosen unsupervised fusion target, not a measured ideal image.

A segmentation network adds task information. The code uses a segmentation model with a main prediction and an auxiliary prediction. With class labels SS, main logits ZZ, auxiliary logits ZaZ_a and hard-example cross-entropy ℓ\ell, the fusion-training semantic term is

Lsem=ℓ(Z,S)+0.1ℓ(Za,S),Ltotal=Lfusion+rLsem.\mathcal L_{\rm sem}=\ell(Z,S)+0.1\ell(Z_a,S),\qquad \mathcal L_{\rm total}=\mathcal L_{\rm fusion}+r\mathcal L_{\rm sem}.

Here rr is the training-round coefficient in the release. Hard-example mining emphasizes difficult labeled pixels. The segmentation model’s weights are frozen during a fusion update, but its computation remains differentiable with respect to the fused input: gradients pass through it into the fusion network. Freezing parameters must not be confused with disabling the entire computation graph.

The outer procedure first obtains fused images, trains or updates segmentation on them, and then trains fusion with semantic feedback. The segmentation-training auxiliary coefficient is 0.75 in the opened code; the fusion-feedback auxiliary coefficient is 0.1. Those are distinct optimization stages. The first fusion round uses the source objective without semantic feedback, while subsequent rounds add the task signal. This staged process is more precise than calling the system one simultaneous joint loss optimized identically throughout.

The semantic labels make this task-aware training supervised with respect to segmentation, even though no ground-truth fused image is required. At deployment the fusion network can run by itself; the training segmentation network is not needed to produce each fused frame.

Results

The official project provides qualitative fusion examples from MFNet and displays segmentation results for infrared, visible and fused image sets. Its evaluation examples distinguish retrained segmentation networks from a pretrained external segmentation system, and also show a pretrained detector. These settings answer different questions: retraining measures what a task model can learn from a representation, while a fixed external model measures transfer to that model.

Run the loss before training a network. The original paper’s numerical tables have not been verified here. This CPU example instead checks the released source-preservation objective [67] on synthetic stripes and a bright square. It scores four fixed candidate images; it does not train or benchmark SeAFusion.

Code · Python · 20 lines
import torch
import torch.nn.functional as F

x = torch.arange(32).reshape(1, 1, 1, 32)
y = (0.2 + 0.4 * ((x // 4) % 2)).expand(1, 1, 32, 32).clone()
i = torch.full_like(y, 0.1)
i[:, :, 8:24, 8:24] = 0.9
kx = torch.tensor([[-1., 0., 1.], [-2., 0., 2.], [-1., 0., 1.]])[None, None]
ky = kx.transpose(-1, -2)

def gradient(z):
    return F.conv2d(z, kx, padding=1).abs() + F.conv2d(z, ky, padding=1).abs()

intensity_target = torch.maximum(y, i)
gradient_target = torch.maximum(gradient(y), gradient(i))
for name, fused in [('visible', y), ('infrared', i),
                    ('mean', (y + i) / 2), ('maximum', intensity_target)]:
    li = F.l1_loss(fused, intensity_target)
    lg = F.l1_loss(gradient(fused), gradient_target)
    print(f'{name:8s} intensity={li:.4f} gradient={lg:.4f} total={li + 10 * lg:.4f}')

The executed output is:

Code · 4 lines
visible  intensity=0.1250 gradient=0.2500 total=2.6250
infrared intensity=0.2250 gradient=0.6867 total=7.0922
mean     intensity=0.1750 gradient=0.5215 total=5.3898
maximum  intensity=0.0000 gradient=0.2625 total=2.6250

The maximum image exactly meets the intensity target, yet its gradient loss remains positive: taking a spatial derivative and taking a maximum do not commute. Visible-only and maximum fusion even tie on the weighted total while retaining different source evidence. This objective therefore cannot certify semantic usefulness. Inspect both modalities and evaluate a task model separately; these numbers establish neither original-paper mIoU nor a network ablation.

Ablations

The design separates three useful tests: remove semantic feedback while retaining the same fusion network and data; remove the gradient branch while retaining decoder capacity as closely as possible; and remove the gradient loss while retaining the architecture. The first tests task feedback, the second architectural gradient features and the third the optimization target. Removing all three at once would not identify which component caused an observed change.

The released training schedule also shows why comparing the first round with a later round is not automatically a clean semantic-loss ablation: the models have undergone different training histories. Matched data, initialization, training budgets and segmentation evaluation are needed. These are design-analysis points, not fabricated numerical ablation outcomes. The guide does not assign unavailable original table values to them.

Conclusion

SeAFusion connects a compact dual-branch fusion model to semantic supervision. The useful distinction is between three signals: learned local image features, fixed source-retention losses and task-network feedback. Together they explain how an image representation can be optimized for both visible evidence and a downstream objective without needing a target fused photograph.

Problems

Semantic supervision can favor the annotated categories and training domain while neglecting unannotated or unfamiliar objects. Maximum-intensity and maximum-gradient targets can retain noise, hot backgrounds and misregistration artifacts. A segmentation gain on one dataset does not establish general perceptual quality, detection benefit or robustness to a different sensor.

For mobile deployment, measure the exported fusion network at the actual resolution and precision; a compact architecture alone does not prove a latency target. Compare TarDAL for detection-aware cooperative learning, and the chapter’s metric counterexample for why both image-level and task-level checks are needed.

References

  1. Y. Liu, X. Chen, Z. Wang, Z. J. Wang, R. K. Ward and X. Wang, “Deep learning for pixel-level image fusion: Recent advances and future prospects,” Information Fusion, vol. 42, pp. 158–173, 2018. doi
  2. J. Liu, G. Wu, Z. Liu, D. Wang, Z. Jiang, L. Ma, W. Zhong, X. Fan and R. Liu, “Infrared and visible image fusion: From data compatibility to task adaption,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 47, no. 4, pp. 2349–2369, 2025. doi arXiv
  3. J. Ma, Y. Ma and C. Li, “Infrared and visible image fusion methods and applications: A survey,” Information Fusion, vol. 45, pp. 153–178, 2019. doi
  4. P. E. Debevec and J. Malik, “Recovering high dynamic range radiance maps from photographs,” in Proc. SIGGRAPH, 1997, pp. 369–378. doi
  5. T. Mertens, J. Kautz and F. Van Reeth, “Exposure fusion: A simple and practical alternative to high dynamic range photography,” Computer Graphics Forum, vol. 28, no. 1, pp. 161–171, 2009. doi
  6. S. Xu, X. Wei, C. Zhang, J. Liu and J. Zhang, “MFFW: A new dataset for multi-focus image fusion,” arXiv:2002.04780, 2020. arXiv
  7. X. Zhang, “Multi-focus image fusion: A benchmark,” arXiv:2005.01116, 2020. arXiv
  8. P. J. Burt and E. H. Adelson, “The Laplacian pyramid as a compact image code,” IEEE Transactions on Communications, vol. 31, no. 4, pp. 532–540, 1983. doi
  9. P. J. Burt and R. J. Kolczynski, “Enhanced image capture through fusion,” in Proc. 4th International Conference on Computer Vision (ICCV), 1993, pp. 173–182. doi
  10. P. J. Burt and E. H. Adelson, “A multiresolution spline with application to image mosaics,” ACM Transactions on Graphics, vol. 2, no. 4, pp. 217–236, 1983. doi
  11. H. Li, B. S. Manjunath and S. K. Mitra, “Multisensor image fusion using the wavelet transform,” Graphical Models and Image Processing, vol. 57, no. 3, pp. 235–245, 1995. doi
  12. S. Li, X. Kang and J. Hu, “Image fusion with guided filtering,” IEEE Transactions on Image Processing, vol. 22, no. 7, pp. 2864–2875, 2013. doi
  13. K. He, J. Sun and X. Tang, “Guided image filtering,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, no. 6, pp. 1397–1409, 2013. doi
  14. B. Yang and S. Li, “Multifocus image fusion and restoration with sparse representation,” IEEE Transactions on Instrumentation and Measurement, vol. 59, no. 4, pp. 884–892, 2010. doi
  15. M. Nejati, S. Samavi and S. Shirani, “Multi-focus image fusion using dictionary-based sparse representation,” Information Fusion, vol. 25, pp. 72–84, 2015. doi
  16. Z. Wang, A. C. Bovik, H. R. Sheikh and E. P. Simoncelli, “Image quality assessment: From error visibility to structural similarity,” IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600–612, 2004. doi
  17. X. Zhang, P. Ye and G. Xiao, “VIFB: A visible and infrared image fusion benchmark,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2020. CVF arXiv
  18. X. Zhang, “Benchmarking and comparing multi-exposure image fusion algorithms,” Information Fusion, vol. 74, pp. 111–131, 2021. doi arXiv
  19. A. M. Eskicioglu and P. S. Fisher, “Image quality measures and their performance,” IEEE Transactions on Communications, vol. 43, no. 12, pp. 2959–2965, 1995. doi
  20. G. Qu, D. Zhang and P. Yan, “Information measure for performance of image fusion,” Electronics Letters, vol. 38, no. 7, pp. 313–315, 2002. doi
  21. M. Hossny, S. Nahavandi and D. Creighton, “Comments on ‘Information measure for performance of image fusion’,” Electronics Letters, vol. 44, no. 18, pp. 1066–1067, 2008. doi
  22. C. S. Xydeas and V. Petrović, “Objective image fusion performance measure,” Electronics Letters, vol. 36, no. 4, pp. 308–309, 2000. doi
  23. G. Piella and H. Heijmans, “A new quality metric for image fusion,” in Proc. IEEE International Conference on Image Processing (ICIP), 2003, vol. 3, pp. III-173–176. doi
  24. Y. Han, Y. Cai, Y. Cao and X. Xu, “A new image fusion performance metric based on visual information fidelity,” Information Fusion, vol. 14, no. 2, pp. 127–135, 2013. doi
  25. K. Ma, K. Zeng and Z. Wang, “Perceptual quality assessment for multi-exposure image fusion,” IEEE Transactions on Image Processing, vol. 24, no. 11, pp. 3345–3356, 2015. doi
  26. Z. Liu, E. Blasch, Z. Xue, J. Zhao, R. Laganiere and W. Wu, “Objective assessment of multiresolution image fusion algorithms for context enhancement in night vision: A comparative study,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 34, no. 1, pp. 94–109, 2012. doi
  27. J. Liu, X. Fan, Z. Huang, G. Wu, R. Liu, W. Zhong and Z. Luo, “Target-aware dual adversarial learning and a multi-scenario multi-modality benchmark to fuse infrared and visible for object detection,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 5802–5811. CVF arXiv M3FD
  28. J. Liu, Z. Liu, G. Wu, L. Ma, R. Liu, W. Zhong, Z. Luo and X. Fan, “Multi-interactive feature learning and a full-time multi-modality benchmark for image fusion and segmentation,” in Proc. IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 8115–8124. CVF arXiv
  29. Y. Zhang, Y. Liu, P. Sun, H. Yan, X. Zhao and L. Zhang, “IFCNN: A general image fusion framework based on convolutional neural network,” Information Fusion, vol. 54, pp. 99–118, 2020. doi
  30. H. Li and X.-J. Wu, “DenseFuse: A fusion approach to infrared and visible images,” IEEE Transactions on Image Processing, vol. 28, no. 5, pp. 2614–2623, 2019. doi arXiv
  31. K. R. Prabhakar, V. S. Srikar and R. V. Babu, “DeepFuse: A deep unsupervised approach for exposure fusion with extreme exposure image pairs,” in Proc. IEEE International Conference on Computer Vision (ICCV), 2017, pp. 4724–4732. doi CVF
  32. H. Xu, J. Ma, J. Jiang, X. Guo and H. Ling, “U2Fusion: A unified unsupervised image fusion network,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 1, pp. 502–518, 2022. doi RoadScene
  33. L. Tang, J. Yuan and J. Ma, “Image fusion in the loop of high-level vision tasks: A semantic-aware real-time infrared and visible image fusion network,” Information Fusion, vol. 82, pp. 28–42, 2022. doi
  34. J. Ma, L. Tang, F. Fan, J. Huang, X. Mei and Y. Ma, “SwinFusion: Cross-domain long-range learning for general image fusion via Swin Transformer,” IEEE/CAA Journal of Automatica Sinica, vol. 9, no. 7, pp. 1200–1217, 2022. doi
  35. X. Zhang and Y. Demiris, “Visible and infrared image fusion using deep learning,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 8, pp. 10535–10554, 2023. doi
  36. Y. Liu, X. Chen, H. Peng and Z. Wang, “Multi-focus image fusion with a deep convolutional neural network,” Information Fusion, vol. 36, pp. 191–207, 2017. doi
  37. K. Ma, Z. Duanmu, H. Zhu, Y. Fang and Z. Wang, “Deep guided learning for fast multi-exposure image fusion,” IEEE Transactions on Image Processing, vol. 29, pp. 2808–2819, 2020. doi
  38. Z. Zhao, H. Bai, J. Zhang, Y. Zhang, S. Xu, Z. Lin, R. Timofte and L. Van Gool, “CDDFuse: Correlation-driven dual-branch feature decomposition for multi-modality image fusion,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 5906–5916. CVF arXiv
  39. J. Ma, W. Yu, P. Liang, C. Li and J. Jiang, “FusionGAN: A generative adversarial network for infrared and visible image fusion,” Information Fusion, vol. 48, pp. 11–26, 2019. doi · public article transcription
  40. H. Bai, J. Zhang, Z. Zhao, Y. Wu, L. Deng, Y. Cui, T. Feng and S. Xu, “Task-driven image fusion with learnable fusion loss,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025, pp. 7457–7468. CVF arXiv
  41. Z. Zhao, H. Bai, Y. Zhu, J. Zhang, S. Xu, Y. Zhang, K. Zhang, D. Meng, R. Timofte and L. Van Gool, “DDFM: Denoising diffusion model for multi-modality image fusion,” in Proc. IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 8082–8093. CVF arXiv
  42. X. Yi, H. Xu, H. Zhang, L. Tang and J. Ma, “Text-IF: Leveraging semantic text guidance for degradation-aware and interactive image fusion,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 27026–27035. CVF arXiv
  43. D. Wang, J. Liu, X. Fan and R. Liu, “Unsupervised misaligned infrared and visible image fusion via cross-modality image generation and registration,” in Proc. International Joint Conference on Artificial Intelligence (IJCAI), 2022, pp. 3508–3515. doi arXiv
  44. L. Tang, Y. Deng, Y. Ma, J. Huang and J. Ma, “SuperFusion: A versatile image registration and fusion network with semantic awareness,” IEEE/CAA Journal of Automatica Sinica, vol. 9, no. 12, pp. 2121–2137, 2022. doi
  45. A. Toet, “The TNO multiband image data collection,” Data in Brief, vol. 15, pp. 249–251, 2017. doi
  46. L. Tang, J. Yuan, H. Zhang, X. Jiang and J. Ma, “PIAFusion: A progressive infrared and visible image fusion network based on illumination aware,” Information Fusion, vol. 83–84, pp. 79–92, 2022. doi MSRS
  47. Q. Ha, K. Watanabe, T. Karasawa, Y. Ushiku and T. Harada, “MFNet: Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes,” in Proc. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2017, pp. 5108–5115. doi
  48. X. Jia, C. Zhu, M. Li, W. Tang and W. Zhou, “LLVIP: A visible-infrared paired dataset for low-light vision,” in Proc. IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, 2021, pp. 3496–3504. CVF arXiv
  49. J. Cai, S. Gu and L. Zhang, “Learning a deep single image contrast enhancer from multi-exposure images,” IEEE Transactions on Image Processing, vol. 27, no. 4, pp. 2049–2062, 2018. doi
  50. X. Zhang, “Deep learning-based multi-focus image fusion: A survey and a comparative study,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 9, 2022. doi
  51. F. Xu, J. Liu, Y. Song, H. Sun and X. Wang, “Multi-exposure image fusion techniques: A comprehensive review,” Remote Sensing, vol. 14, no. 3, art. 771, 2022. doi
  52. Z. Zhao, H. Bai, J. Zhang, Y. Zhang, K. Zhang, S. Xu, D. Chen, R. Timofte and L. Van Gool, “Equivariant multi-modality image fusion,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 25912–25921. CVF arXiv
  53. L. Tang, Y. Wang, M. Gong, Z. Li, Y. Deng, X. Yi, C. Li, H. Xu, H. Zhang and J. Ma, “VideoFusion: A spatio-temporal collaborative network for multi-modal video fusion,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026, pp. 19559–19569. CVF
  54. N. Yokoya, C. Grohnfeldt and J. Chanussot, “Hyperspectral and multispectral data fusion: A comparative review of the recent literature,” IEEE Geoscience and Remote Sensing Magazine, 2017. doi
  55. H. R. Sheikh and A. C. Bovik, “Image information and visual quality,” IEEE Transactions on Image Processing, 2006. doi
  56. H. Xu et al., “U2Fusion,” official implementation accompanying the TPAMI paper, 2022. code
  57. K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” ICLR, 2015. arXiv
  58. J. Kirkpatrick et al., “Overcoming catastrophic forgetting in neural networks,” Proceedings of the National Academy of Sciences, 2017. doi
  59. Y. Zhang et al., “IFCNN,” official implementation and errata accompanying the Information Fusion paper, 2020. code
  60. K. He, X. Zhang, S. Ren and J. Sun, “Deep residual learning for image recognition,” CVPR, 2016. CVF
  61. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li and L. Fei-Fei, “ImageNet: A large-scale hierarchical image database,” CVPR, 2009. doi
  62. N. Silberman, D. Hoiem, P. Kohli and R. Fergus, “Indoor segmentation and support inference from RGBD images,” ECCV, 2012. project
  63. P. Dhariwal and A. Nichol, “Diffusion models beat GANs on image synthesis,” NeurIPS, 2021. arXiv
  64. J. Liu et al., “TarDAL,” official implementation accompanying the CVPR paper, 2022. code
  65. G. Jocher et al., “YOLOv5,” Ultralytics object-detection software, 2020 onward. code
  66. J. Ma et al., “FusionGAN,” official implementation accompanying the Information Fusion paper, 2019. code
  67. L. Tang, J. Yuan and J. Ma, “SeAFusion,” official implementation accompanying the Information Fusion paper, 2022. code
  68. T. Mertens, J. Kautz and F. Van Reeth, “Exposure Fusion,” in Proc. Pacific Graphics, 2007. paper · author project