完整搜尋頁面 →

建構與還原拉普拉斯金字塔

選讀・DLCV 10・影像融合:互補來源與有用輸出 的補充資料(第 2/6 則)。主線不依賴這則補充;引用編號對應本頁底部的參考文獻。

導覽:本頁與本系列: 導覽:本頁與本系列

本頁內容

補充資料 · 6

  1. 散焦擴散效應
  2. 建構與還原拉普拉斯金字塔
  3. 稀疏表示融合
  4. 互資訊作為融合指標
  5. 代表性融合網路的訓練訊號
  6. VIF 與來源參考的融合分數

論文導讀 · 11

  1. 曝光融合(Mertens、Kautz 與 Van Reeth)
  2. VIFB:可見光與紅外線影像融合基準
  3. Liu 等人的紅外線與可見光融合綜述(TPAMI 2025)
  4. DenseFuse:學習重建,在特徵空間選擇融合規則
  5. DDFM:以概似修正擴散先驗
  6. FusionGAN:熱強度與對抗式可見光先驗
  7. IFCNN:支援可變輸入數的監督式融合
  8. U2Fusion:跨融合任務共用一個無監督模型
  9. SwinFusion:來源內與來源間的注意力
  10. SeAFusion:讓語意分割影響融合影像
  11. TarDAL:同時為外觀與物件偵測融合

電腦視覺深度學習

  1. 02 · 深度影像分類
  2. 03 · 物件偵測:從 R-CNN 到 DETR
  3. 09 · 深度影像復原:低階視覺
查看完整課程樹 →

這一則把全篇用到的拉普拉斯金字塔寫完整:濾波器、大小處理、為什麼還原是精確的,以及實際實作中的陷阱。

兩個運算子。 REDUCE⁡\operatorname{REDUCE} 用一個小的低通核 kk 對影像做卷積,並每隔一列、每隔一行保留一個;EXPAND⁡\operatorname{EXPAND} 在樣本之間插入零,用 4k4k 做卷積(乘上 4 是為了補回插入零所損失的平均亮度),並回傳兩倍大的影像。OpenCV 的 pyrDown 與 pyrUp 使用可分離的核 k=1256 vv⊤k = \tfrac{1}{256}\,\mathbf{v}\mathbf{v}^\top,其中 v=(1,4,6,4,1)⊤\mathbf{v} = (1, 4, 6, 4, 1)^\top,是高斯的二項式近似;你可以用一個脈衝來檢查:

程式碼 · Python · 10 行
import numpy as np
import cv2

small = np.zeros((5, 5), np.float32)
small[2, 2] = 64.0
print(cv2.pyrUp(small)[2:7, 2:7])              # zeros inserted, then 4 x the kernel: 64 * 4 * v v^T / 256
impulse = np.zeros((9, 9), np.float32)
impulse[4, 4] = 256.0
print(cv2.pyrDown(impulse)[1:4, 1:4])          # blur, then keep every 2nd pixel: taps at offsets 0 and +-2
print(cv2.pyrDown(np.zeros((7, 5), np.float32)).shape)   # odd sizes round up: (7, 5) -> (4, 3)

輸出

[[ 1.  4.  6.  4.  1.]
 [ 4. 16. 24. 16.  4.]
 [ 6. 24. 36. 24.  6.]
 [ 4. 16. 24. 16.  4.]
 [ 1.  4.  6.  4.  1.]]
[[ 1.  6.  1.]
 [ 6. 36.  6.]
 [ 1.  6.  1.]]
(4, 3)

大小。 H×WH \times W 的一層縮小後是 ⌈H/2⌉×⌈W/2⌉\lceil H/2 \rceil \times \lceil W/2 \rceil。對奇數尺寸,除非告訴 EXPAND⁡\operatorname{EXPAND} 目標尺寸,否則較小一層經它放大後會多出一個像素,這就是本篇程式碼會把 dstsize 傳給 pyrUp 的原因(輸出的最後一行顯示了 ⌈⋅/2⌉\lceil \cdot / 2 \rceil 規則)。忘記這一點是金字塔程式碼的常見錯誤:減法 Gℓ−EXPAND⁡(Gℓ+1)G_\ell - \operatorname{EXPAND}(G_{\ell+1}) 會失敗,或悄悄地廣播。

為什麼還原是精確的。 依定義 Lℓ=Gℓ−EXPAND⁡(Gℓ+1)L_\ell = G_\ell - \operatorname{EXPAND}(G_{\ell+1}),所以 Gℓ=Lℓ+EXPAND⁡(Gℓ+1)G_\ell = L_\ell + \operatorname{EXPAND}(G_{\ell+1})。從 GN=LNG_N = L_N 出發,對 ℓ=N−1,…,0\ell = N-1, \dots, 0 套用這個恆等式,就會重建每個 GℓG_\ell,最後得到 G0=IG_0 = I。這裡完全沒有用到核的任何性質:金字塔儲存的是 EXPAND⁡\operatorname{EXPAND} 的預測誤差,所以任何 EXPAND⁡\operatorname{EXPAND} 都可以。真正取決於核的,是各層的意義:有了好的低通核,LℓL_\ell 就是對某一種大小的特徵有響應的帶通影像,取絕對值最大規則才因此有意義。

融合後的金字塔不是任何影像的金字塔。 融合之後,這疊 {LℓF}\{L^F_\ell\} 一般來說不是它自己還原結果的拉普拉斯金字塔:還原後再重新分解,會得到不同的層。這無妨(我們只還原一次),但這也是融合結果可能含有小光暈的原因之一:在相鄰層獨立選出的係數,彼此不一定一致。

儲存量。 第 ℓ\ell 層約有 HW/4ℓHW/4^\ell 個值,所以整座金字塔約有 HW(1+14+116+… )≈43HWHW(1 + \tfrac14 + \tfrac1{16} + \dots) \approx \tfrac43 HW 個值:拉普拉斯金字塔略微過完備,而 IP 07 的正交小波轉換是臨界取樣的(剛好 HWHW 個值),但把每個尺度拆成三個方向頻帶。

該用幾層。 每一層使尺寸減半,所以最多約能放進 log⁡2min⁡(H,W)\log_2 \min(H, W) 層。對於把粗略基底取平均的融合規則,層數越多,平均就發生在越粗的尺度,這會減少低頻瑕疵;Mertens 等人指出,較高的金字塔可部分修正他們那種虛假的低頻亮度變化,但高度受限於濾波器相對於影像的大小 [5]。練習 4 在玩具範例上量測這個效應。

陷阱。

  • 資料型別。 拉普拉斯層有正負號。用 uint8 陣列建構會截斷或迴繞負值;請先轉成 float32,如本篇程式碼所做的。
  • 色彩。 每個通道分別分解,但所有通道要用同一個決策(或同一張權重圖),否則各通道可能來自不同的來源,產生彩色邊紋。
  • 邊界。 靠近影像邊界的卷積需要延伸規則(OpenCV 預設以邊界像素為軸反射影像)。不同的規則在邊界附近給出略微不同的係數,這是不同實作之間出現小差異的常見來源。

參考文獻

  1. Y. Liu, X. Chen, Z. Wang, Z. J. Wang, R. K. Ward and X. Wang, “Deep learning for pixel-level image fusion: Recent advances and future prospects,” Information Fusion, vol. 42, pp. 158–173, 2018. doi
  2. J. Liu, G. Wu, Z. Liu, D. Wang, Z. Jiang, L. Ma, W. Zhong, X. Fan and R. Liu, “Infrared and visible image fusion: From data compatibility to task adaption,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 47, no. 4, pp. 2349–2369, 2025. doi arXiv
  3. J. Ma, Y. Ma and C. Li, “Infrared and visible image fusion methods and applications: A survey,” Information Fusion, vol. 45, pp. 153–178, 2019. doi
  4. P. E. Debevec and J. Malik, “Recovering high dynamic range radiance maps from photographs,” in Proc. SIGGRAPH, 1997, pp. 369–378. doi
  5. T. Mertens, J. Kautz and F. Van Reeth, “Exposure fusion: A simple and practical alternative to high dynamic range photography,” Computer Graphics Forum, vol. 28, no. 1, pp. 161–171, 2009. doi
  6. S. Xu, X. Wei, C. Zhang, J. Liu and J. Zhang, “MFFW: A new dataset for multi-focus image fusion,” arXiv:2002.04780, 2020. arXiv
  7. X. Zhang, “Multi-focus image fusion: A benchmark,” arXiv:2005.01116, 2020. arXiv
  8. P. J. Burt and E. H. Adelson, “The Laplacian pyramid as a compact image code,” IEEE Transactions on Communications, vol. 31, no. 4, pp. 532–540, 1983. doi
  9. P. J. Burt and R. J. Kolczynski, “Enhanced image capture through fusion,” in Proc. 4th International Conference on Computer Vision (ICCV), 1993, pp. 173–182. doi
  10. P. J. Burt and E. H. Adelson, “A multiresolution spline with application to image mosaics,” ACM Transactions on Graphics, vol. 2, no. 4, pp. 217–236, 1983. doi
  11. H. Li, B. S. Manjunath and S. K. Mitra, “Multisensor image fusion using the wavelet transform,” Graphical Models and Image Processing, vol. 57, no. 3, pp. 235–245, 1995. doi
  12. S. Li, X. Kang and J. Hu, “Image fusion with guided filtering,” IEEE Transactions on Image Processing, vol. 22, no. 7, pp. 2864–2875, 2013. doi
  13. K. He, J. Sun and X. Tang, “Guided image filtering,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, no. 6, pp. 1397–1409, 2013. doi
  14. B. Yang and S. Li, “Multifocus image fusion and restoration with sparse representation,” IEEE Transactions on Instrumentation and Measurement, vol. 59, no. 4, pp. 884–892, 2010. doi
  15. M. Nejati, S. Samavi and S. Shirani, “Multi-focus image fusion using dictionary-based sparse representation,” Information Fusion, vol. 25, pp. 72–84, 2015. doi
  16. Z. Wang, A. C. Bovik, H. R. Sheikh and E. P. Simoncelli, “Image quality assessment: From error visibility to structural similarity,” IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600–612, 2004. doi
  17. X. Zhang, P. Ye and G. Xiao, “VIFB: A visible and infrared image fusion benchmark,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2020. CVF arXiv
  18. X. Zhang, “Benchmarking and comparing multi-exposure image fusion algorithms,” Information Fusion, vol. 74, pp. 111–131, 2021. doi arXiv
  19. A. M. Eskicioglu and P. S. Fisher, “Image quality measures and their performance,” IEEE Transactions on Communications, vol. 43, no. 12, pp. 2959–2965, 1995. doi
  20. G. Qu, D. Zhang and P. Yan, “Information measure for performance of image fusion,” Electronics Letters, vol. 38, no. 7, pp. 313–315, 2002. doi
  21. M. Hossny, S. Nahavandi and D. Creighton, “Comments on ‘Information measure for performance of image fusion’,” Electronics Letters, vol. 44, no. 18, pp. 1066–1067, 2008. doi
  22. C. S. Xydeas and V. Petrović, “Objective image fusion performance measure,” Electronics Letters, vol. 36, no. 4, pp. 308–309, 2000. doi
  23. G. Piella and H. Heijmans, “A new quality metric for image fusion,” in Proc. IEEE International Conference on Image Processing (ICIP), 2003, vol. 3, pp. III-173–176. doi
  24. Y. Han, Y. Cai, Y. Cao and X. Xu, “A new image fusion performance metric based on visual information fidelity,” Information Fusion, vol. 14, no. 2, pp. 127–135, 2013. doi
  25. K. Ma, K. Zeng and Z. Wang, “Perceptual quality assessment for multi-exposure image fusion,” IEEE Transactions on Image Processing, vol. 24, no. 11, pp. 3345–3356, 2015. doi
  26. Z. Liu, E. Blasch, Z. Xue, J. Zhao, R. Laganiere and W. Wu, “Objective assessment of multiresolution image fusion algorithms for context enhancement in night vision: A comparative study,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 34, no. 1, pp. 94–109, 2012. doi
  27. J. Liu, X. Fan, Z. Huang, G. Wu, R. Liu, W. Zhong and Z. Luo, “Target-aware dual adversarial learning and a multi-scenario multi-modality benchmark to fuse infrared and visible for object detection,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 5802–5811. CVF arXiv M3FD
  28. J. Liu, Z. Liu, G. Wu, L. Ma, R. Liu, W. Zhong, Z. Luo and X. Fan, “Multi-interactive feature learning and a full-time multi-modality benchmark for image fusion and segmentation,” in Proc. IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 8115–8124. CVF arXiv
  29. Y. Zhang, Y. Liu, P. Sun, H. Yan, X. Zhao and L. Zhang, “IFCNN: A general image fusion framework based on convolutional neural network,” Information Fusion, vol. 54, pp. 99–118, 2020. doi
  30. H. Li and X.-J. Wu, “DenseFuse: A fusion approach to infrared and visible images,” IEEE Transactions on Image Processing, vol. 28, no. 5, pp. 2614–2623, 2019. doi arXiv
  31. K. R. Prabhakar, V. S. Srikar and R. V. Babu, “DeepFuse: A deep unsupervised approach for exposure fusion with extreme exposure image pairs,” in Proc. IEEE International Conference on Computer Vision (ICCV), 2017, pp. 4724–4732. doi CVF
  32. H. Xu, J. Ma, J. Jiang, X. Guo and H. Ling, “U2Fusion: A unified unsupervised image fusion network,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 1, pp. 502–518, 2022. doi RoadScene
  33. L. Tang, J. Yuan and J. Ma, “Image fusion in the loop of high-level vision tasks: A semantic-aware real-time infrared and visible image fusion network,” Information Fusion, vol. 82, pp. 28–42, 2022. doi
  34. J. Ma, L. Tang, F. Fan, J. Huang, X. Mei and Y. Ma, “SwinFusion: Cross-domain long-range learning for general image fusion via Swin Transformer,” IEEE/CAA Journal of Automatica Sinica, vol. 9, no. 7, pp. 1200–1217, 2022. doi
  35. X. Zhang and Y. Demiris, “Visible and infrared image fusion using deep learning,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 8, pp. 10535–10554, 2023. doi
  36. Y. Liu, X. Chen, H. Peng and Z. Wang, “Multi-focus image fusion with a deep convolutional neural network,” Information Fusion, vol. 36, pp. 191–207, 2017. doi
  37. K. Ma, Z. Duanmu, H. Zhu, Y. Fang and Z. Wang, “Deep guided learning for fast multi-exposure image fusion,” IEEE Transactions on Image Processing, vol. 29, pp. 2808–2819, 2020. doi
  38. Z. Zhao, H. Bai, J. Zhang, Y. Zhang, S. Xu, Z. Lin, R. Timofte and L. Van Gool, “CDDFuse: Correlation-driven dual-branch feature decomposition for multi-modality image fusion,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 5906–5916. CVF arXiv
  39. J. Ma, W. Yu, P. Liang, C. Li and J. Jiang, “FusionGAN: A generative adversarial network for infrared and visible image fusion,” Information Fusion, vol. 48, pp. 11–26, 2019. doi · public article transcription
  40. H. Bai, J. Zhang, Z. Zhao, Y. Wu, L. Deng, Y. Cui, T. Feng and S. Xu, “Task-driven image fusion with learnable fusion loss,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025, pp. 7457–7468. CVF arXiv
  41. Z. Zhao, H. Bai, Y. Zhu, J. Zhang, S. Xu, Y. Zhang, K. Zhang, D. Meng, R. Timofte and L. Van Gool, “DDFM: Denoising diffusion model for multi-modality image fusion,” in Proc. IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 8082–8093. CVF arXiv
  42. X. Yi, H. Xu, H. Zhang, L. Tang and J. Ma, “Text-IF: Leveraging semantic text guidance for degradation-aware and interactive image fusion,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 27026–27035. CVF arXiv
  43. D. Wang, J. Liu, X. Fan and R. Liu, “Unsupervised misaligned infrared and visible image fusion via cross-modality image generation and registration,” in Proc. International Joint Conference on Artificial Intelligence (IJCAI), 2022, pp. 3508–3515. doi arXiv
  44. L. Tang, Y. Deng, Y. Ma, J. Huang and J. Ma, “SuperFusion: A versatile image registration and fusion network with semantic awareness,” IEEE/CAA Journal of Automatica Sinica, vol. 9, no. 12, pp. 2121–2137, 2022. doi
  45. A. Toet, “The TNO multiband image data collection,” Data in Brief, vol. 15, pp. 249–251, 2017. doi
  46. L. Tang, J. Yuan, H. Zhang, X. Jiang and J. Ma, “PIAFusion: A progressive infrared and visible image fusion network based on illumination aware,” Information Fusion, vol. 83–84, pp. 79–92, 2022. doi MSRS
  47. Q. Ha, K. Watanabe, T. Karasawa, Y. Ushiku and T. Harada, “MFNet: Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes,” in Proc. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2017, pp. 5108–5115. doi
  48. X. Jia, C. Zhu, M. Li, W. Tang and W. Zhou, “LLVIP: A visible-infrared paired dataset for low-light vision,” in Proc. IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, 2021, pp. 3496–3504. CVF arXiv
  49. J. Cai, S. Gu and L. Zhang, “Learning a deep single image contrast enhancer from multi-exposure images,” IEEE Transactions on Image Processing, vol. 27, no. 4, pp. 2049–2062, 2018. doi
  50. X. Zhang, “Deep learning-based multi-focus image fusion: A survey and a comparative study,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 9, 2022. doi
  51. F. Xu, J. Liu, Y. Song, H. Sun and X. Wang, “Multi-exposure image fusion techniques: A comprehensive review,” Remote Sensing, vol. 14, no. 3, art. 771, 2022. doi
  52. Z. Zhao, H. Bai, J. Zhang, Y. Zhang, K. Zhang, S. Xu, D. Chen, R. Timofte and L. Van Gool, “Equivariant multi-modality image fusion,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 25912–25921. CVF arXiv
  53. L. Tang, Y. Wang, M. Gong, Z. Li, Y. Deng, X. Yi, C. Li, H. Xu, H. Zhang and J. Ma, “VideoFusion: A spatio-temporal collaborative network for multi-modal video fusion,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026, pp. 19559–19569. CVF
  54. N. Yokoya, C. Grohnfeldt and J. Chanussot, “Hyperspectral and multispectral data fusion: A comparative review of the recent literature,” IEEE Geoscience and Remote Sensing Magazine, 2017. doi
  55. H. R. Sheikh and A. C. Bovik, “Image information and visual quality,” IEEE Transactions on Image Processing, 2006. doi
  56. H. Xu et al., “U2Fusion,” official implementation accompanying the TPAMI paper, 2022. code
  57. K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” ICLR, 2015. arXiv
  58. J. Kirkpatrick et al., “Overcoming catastrophic forgetting in neural networks,” Proceedings of the National Academy of Sciences, 2017. doi
  59. Y. Zhang et al., “IFCNN,” official implementation and errata accompanying the Information Fusion paper, 2020. code
  60. K. He, X. Zhang, S. Ren and J. Sun, “Deep residual learning for image recognition,” CVPR, 2016. CVF
  61. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li and L. Fei-Fei, “ImageNet: A large-scale hierarchical image database,” CVPR, 2009. doi
  62. N. Silberman, D. Hoiem, P. Kohli and R. Fergus, “Indoor segmentation and support inference from RGBD images,” ECCV, 2012. project
  63. P. Dhariwal and A. Nichol, “Diffusion models beat GANs on image synthesis,” NeurIPS, 2021. arXiv
  64. J. Liu et al., “TarDAL,” official implementation accompanying the CVPR paper, 2022. code
  65. G. Jocher et al., “YOLOv5,” Ultralytics object-detection software, 2020 onward. code
  66. J. Ma et al., “FusionGAN,” official implementation accompanying the Information Fusion paper, 2019. code
  67. L. Tang, J. Yuan and J. Ma, “SeAFusion,” official implementation accompanying the Information Fusion paper, 2022. code
  68. T. Mertens, J. Kautz and F. Van Reeth, “Exposure Fusion,” in Proc. Pacific Graphics, 2007. paper · author project