完整搜尋頁面 →

Chen 等人:Simple Baselines for Image Restoration(NAFNet,ECCV 2022)

選讀・DLCV 01B・網路的積木與組合:分類與影像復原網路 的論文導讀(第 1/4 篇)。引用編號對應本頁底部的參考文獻。

導覽:本頁與本系列: 導覽:本頁與本系列

本頁內容

補充資料 · 5

  1. CBAM 的通道閘與空間閘
  2. 一維轉置卷積的相位和
  3. 經過上取樣與跳接的感受野
  4. ConvNeXt 路線圖,一步一步
  5. RegNet 的量化線性寬度

論文導讀 · 4

  1. Chen 等人:Simple Baselines for Image Restoration(NAFNet,ECCV 2022)
  2. Radosavovic 等人:Designing Network Design Spaces(RegNet,CVPR 2020)
  3. Chen 等人:A Comparative Study of Image Restoration Networks for General Backbone Network Design(X-Restormer,ECCV 2024)
  4. RepVGG:訓練用分支,部署用直線網路

電腦視覺深度學習

  1. 02 · 深度影像分類
  2. 03 · 物件偵測:從 R-CNN 到 DETR
  3. 09 · 深度影像復原:低階視覺
查看完整課程樹 →

挑戰

Chen 等人 [19] 問的是:影像修復真的需要愈來愈複雜的區塊嗎?他們保留單階段 U-Net,先以受控增量建立有競爭力的基準,再移除能被簡單閘門取代的元件。這是在區分整體架構與區塊設計的貢獻,不是宣稱所有非線性計算都不需要。

方法

由左至右比較基準與 NAF 區塊:乘法取代 GELU,簡化通道注意力仍保留依輸入改變的閘門。
原論文圖 3 — 由左至右比較基準與 NAF 區塊:乘法取代 GELU,簡化通道注意力仍保留依輸入改變的閘門。

圖片來源: [19], PDF 第 6 頁. 原始 PDF.

先建立可信的基準,再簡化。 從卷積網路出發,逐步加入層正規化、將 ReLU 換為 GELU,並加入通道注意力。穩定的基準才能用來評估簡化是否有效;正規化實驗同時改變學習率,歸因時必須考慮這點。

以乘法取代具名啟用函數。 逐點擴展與逐通道卷積後,把偶數通道的特徵切成相同大小的 U,VU,V,SimpleGate 定義為:

SG⁡([U,V])=U⊙V.\operatorname{SG}([U,V])=U\odot V.

[U,V][U,V] 表示通道串接,⊙\odot 為逐元素乘法;輸出通道數減半。這仍是非線性運算:兩條支路都放大 aa 倍,乘積會放大 a2a^2 倍。「Nonlinear Activation Free」指移除 GELU 等傳統啟用函數,並非把網路變成線性濾波器。

通道注意力也能簡化。 對閘控後的特徵 FF,簡化通道注意力可寫成:

SCA⁡(F)=F⊙W GAP⁡(F).\operatorname{SCA}(F)=F\odot W\,\operatorname{GAP}(F).

GAP⁡\operatorname{GAP} 對空間位置取平均,WW 是學得的通道投影,所得通道向量廣播到各空間位置。移除原本注意力的瓶頸與啟用函數後,仍保留依輸入而變的乘法閘門。

NAF 區塊的第一條殘差路徑包含正規化、擴展、逐通道濾波、SimpleGate、SCA 與投影;第二條正規化前饋路徑再使用 SimpleGate 與殘差相加。可學習的殘差尺度穩定初始映射。編碼器—解碼器仍負責跨解析度傳遞資訊,簡單區塊不能取代跳接與多尺度路徑。

讀圖與選用判斷。 沿著區塊追蹤特徵張量:正規化整理輸入,空間卷積混合鄰近資訊,SimpleGate 將兩組通道相乘,再進行通道加權與殘差相加。即使沒有具名的活化函數,乘法仍使映射具有非線性。評估替換區塊時,要固定修復任務、寬度、訓練流程與推論轉換;否則無法把進步歸因於移除活化函數。它適合有成對訓練資料時作為精簡的學習式基準,但不會估計物理模糊核。

結果

Table 6 的完整去雜訊模型在 SIDD 達 40.30 dB/0.962 SSIM,成本 65G MACs;較強基準顯示相同品質,但需要 84G MACs。這些完整模型結果須與較小的消融設定分開。

Table 4 的最終去模糊設定在 GoPro 達 33.69 dB/0.967 SSIM,其中包含測試時局部轉換;不能把全部進步都歸給區塊本身。

消融實驗

Table 1 累積建立基準:PlainNet 在學習率 10−410^{-4} 時 GoPro PSNR 為 28.51,改成 10−310^{-3} 的訓練不穩定;正規化讓較大學習率可行,達 31.90。這組比較同時改變訓練穩定性與最佳化設定。

Table 2 的 GoPro 基準為 32.35 dB,只換 SimpleGate 為 32.76、只換 SCA 為 32.54、兩者一起換為 32.85;SIDD 對應 39.85、39.93、39.95、39.96。不同工作對元件的受益幅度不同。

Table 5 重新在閘門內加入啟用函數,有些設定讓 SIDD 略升、GoPro 卻下降;支持簡單設計有用,未證明某種啟用函數在所有工作都最佳。Table 3 也顯示增加深度會增加延遲,品質並非全面單調上升。

結論

NAFNet 的論點是:有良好基準、乘法閘門與受控消融的樸素架構,可以匹敵更複雜的修復模型。必須區分「拿掉具名啟用函數」與「拿掉非線性特徵互動」。

限制與問題

簡化區塊不會消除訓練資料、記憶體與全域池化的需求。區塊訓練與完整影像推論的池化統計可能不同,因此論文評估局部轉換。成對去雜訊/去模糊結果不能直接外推到任意真實退化;相同運算量也不等於相同裝置延遲。

參考文獻

  1. K. He, X. Zhang, S. Ren and J. Sun, “Deep Residual Learning for Image Recognition,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. CVF · arXiv

  2. Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell and S. Xie, “A ConvNet for the 2020s,” in Proc. IEEE/CVF CVPR, 2022. CVF · arXiv

  3. A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer et al., “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale,” in International Conference on Learning Representations (ICLR), 2021. arXiv

  4. J. Liang, J. Cao, G. Sun, K. Zhang, L. Van Gool and R. Timofte, “SwinIR: Image Restoration Using Swin Transformer,” in Proc. IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), 2021. CVF · arXiv

  5. T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan and S. Belongie, “Feature Pyramid Networks for Object Detection,” in Proc. IEEE CVPR, 2017. CVF · arXiv

  6. Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin and B. Guo, “Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows,” in Proc. IEEE/CVF International Conference on Computer Vision (ICCV), 2021. CVF · arXiv

  7. O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy et al., “ImageNet Large Scale Visual Recognition Challenge,” International Journal of Computer Vision, vol. 115, no. 3, pp. 211–252, 2015. doi

  8. W. Yu, M. Luo, P. Zhou, C. Si, Y. Zhou, X. Wang, J. Feng and S. Yan, “MetaFormer Is Actually What You Need for Vision,” in Proc. IEEE/CVF CVPR, 2022. CVF · arXiv

  9. A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto and H. Adam, “MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications,” arXiv:1704.04861, 2017. arXiv

  10. M. Sandler, A. Howard, M. Zhu, A. Zhmoginov and L.-C. Chen, “MobileNetV2: Inverted Residuals and Linear Bottlenecks,” in Proc. IEEE/CVF CVPR, 2018. CVF · arXiv

  11. K. He, X. Zhang, S. Ren and J. Sun, “Identity Mappings in Deep Residual Networks,” in Proc. European Conference on Computer Vision (ECCV), LNCS, Springer, 2016. doi · arXiv

  12. B. Lim, S. Son, H. Kim, S. Nah and K. M. Lee, “Enhanced Deep Residual Networks for Single Image Super-Resolution,” in Proc. IEEE CVPR Workshops (CVPRW), 2017. CVF · arXiv

  13. H. Touvron, M. Cord, A. Sablayrolles, G. Synnaeve and H. Jégou, “Going Deeper With Image Transformers,” in Proc. IEEE/CVF ICCV, 2021. CVF

  14. L. Chen, X. Chu, X. Zhang and J. Sun, NAFNet official implementation, basicsr/models/archs/NAFNet_arch.py, GitHub (megvii-research/NAFNet), accessed 2026. code

  15. Y. Zhang, K. Li, K. Li, L. Wang, B. Zhong and Y. Fu, “Image Super-Resolution Using Very Deep Residual Channel Attention Networks,” in Proc. ECCV, 2018. CVF · arXiv

  16. I. Radosavovic, R. P. Kosaraju, R. Girshick, K. He and P. Dollár, “Designing Network Design Spaces,” in Proc. IEEE/CVF CVPR, 2020. CVF · arXiv

  17. J. Hu, L. Shen and G. Sun, “Squeeze-and-Excitation Networks,” in Proc. IEEE/CVF CVPR, 2018. CVF · arXiv

  18. S. Woo, J. Park, J.-Y. Lee and I. S. Kweon, “CBAM: Convolutional Block Attention Module,” in Proc. ECCV, 2018. CVF · arXiv

  19. L. Chen, X. Chu, X. Zhang and J. Sun, “Simple Baselines for Image Restoration,” in Proc. ECCV, LNCS, Springer, 2022. doi · arXiv

  20. M.-H. Guo, T.-X. Xu, J.-J. Liu, Z.-N. Liu, P.-T. Jiang, T.-J. Mu, S.-H. Zhang, R. R. Martin, M.-M. Cheng and S.-M. Hu, “Attention Mechanisms in Computer Vision: A Survey,” Computational Visual Media, vol. 8, no. 3, pp. 331–368, 2022. doi · arXiv

  21. S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan and M.-H. Yang, “Restormer: Efficient Transformer for High-Resolution Image Restoration,” in Proc. IEEE/CVF CVPR, 2022. CVF · arXiv

  22. X. Chen, Z. Li, Y. Pu, Y. Liu, J. Zhou, Y. Qiao and C. Dong, “A Comparative Study of Image Restoration Networks for General Backbone Network Design,” in Proc. ECCV 2024, LNCS, Springer, 2025. doi · arXiv

  23. S. Ioffe and C. Szegedy, “Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift,” in Proc. International Conference on Machine Learning (ICML), PMLR 37, 2015. PMLR

  24. J. L. Ba, J. R. Kiros and G. E. Hinton, “Layer Normalization,” arXiv:1607.06450, 2016. arXiv

  25. D. Ulyanov, A. Vedaldi and V. Lempitsky, “Instance Normalization: The Missing Ingredient for Fast Stylization,” arXiv:1607.08022, 2016. arXiv

  26. Y. Wu and K. He, “Group Normalization,” in Proc. ECCV, 2018. CVF · arXiv

  27. C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang and W. Shi, “Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network,” in Proc. IEEE CVPR, 2017. CVF

  28. X. Wang, K. Yu, S. Wu, J. Gu, Y. Liu, C. Dong, Y. Qiao and C. C. Loy, “ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks,” in Proc. ECCV Workshops, 2018. CVF · arXiv

  29. K. Zhang, W. Zuo, Y. Chen, D. Meng and L. Zhang, “Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising,” IEEE Transactions on Image Processing, vol. 26, no. 7, pp. 3142–3155, 2017. doi · arXiv

  30. Q. Huynh-Thu and M. Ghanbari, “Scope of Validity of PSNR in Image/Video Quality Assessment,” Electronics Letters, vol. 44, no. 13, pp. 800–801, 2008. doi

  31. A. Abdelhamed, S. Lin and M. S. Brown, “A High-Quality Denoising Dataset for Smartphone Cameras,” in Proc. IEEE/CVF CVPR, 2018. CVF

  32. S. Nah, T. H. Kim and K. M. Lee, “Deep Multi-Scale Convolutional Neural Network for Dynamic Scene Deblurring,” in Proc. IEEE CVPR, 2017. CVF

  33. N. Shazeer, “GLU Variants Improve Transformer,” arXiv:2002.05202, 2020. arXiv

  34. Y. N. Dauphin, A. Fan, M. Auli and D. Grangier, “Language Modeling with Gated Convolutional Networks,” in Proc. ICML, PMLR 70, 2017. PMLR

  35. D. Hendrycks and K. Gimpel, “Gaussian Error Linear Units (GELUs),” arXiv:1606.08415, 2016. arXiv

  36. W. Shi, J. Caballero, F. Huszár, J. Totz, A. P. Aitken, R. Bishop, D. Rueckert and Z. Wang, “Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network,” in Proc. IEEE CVPR, 2016. CVF · arXiv

  37. A. Odena, V. Dumoulin and C. Olah, “Deconvolution and Checkerboard Artifacts,” Distill, vol. 1, no. 10, 2016. Distill · doi

  38. A. Aitken, C. Ledig, L. Theis, J. Caballero, Z. Wang and W. Shi, “Checkerboard Artifact Free Sub-Pixel Convolution: A Note on Sub-Pixel Convolution, Resize Convolution and Convolution Resize,” arXiv:1707.02937, 2017. arXiv

  39. O. Ronneberger, P. Fischer and T. Brox, “U-Net: Convolutional Networks for Biomedical Image Segmentation,” in Proc. Medical Image Computing and Computer-Assisted Intervention (MICCAI), LNCS, Springer, pp. 234–241, 2015. doi · arXiv

  40. G. Huang, Z. Liu, L. van der Maaten and K. Q. Weinberger, “Densely Connected Convolutional Networks,” in Proc. IEEE CVPR, 2017. CVF

  41. L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy and A. L. Yuille, “DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 40, no. 4, pp. 834–848, 2018. doi · arXiv

  42. W. Luo, Y. Li, R. Urtasun and R. Zemel, “Understanding the Effective Receptive Field in Deep Convolutional Neural Networks,” in Advances in Neural Information Processing Systems (NeurIPS), 2016. NeurIPS · arXiv

  43. J. Kim, J. K. Lee and K. M. Lee, “Accurate Image Super-Resolution Using Very Deep Convolutional Networks,” in Proc. IEEE CVPR, 2016. CVF · arXiv

  44. H. Guo, J. Li, T. Dai, Z. Ouyang, X. Ren and S.-T. Xia, “MambaIR: A Simple Baseline for Image Restoration with State-Space Model,” in Proc. ECCV 2024, LNCS, Springer, 2025. doi · arXiv

  45. ITU-T, “Recommendation T.81: Information technology – Digital compression and coding of continuous-tone still images – Requirements and guidelines” (the JPEG standard), 1992. ITU

  46. T. Elsken, J. H. Metzen and F. Hutter, “Neural Architecture Search: A Survey,” Journal of Machine Learning Research, vol. 20, 2019. JMLR · arXiv

  47. A. Khan, A. Sohail, U. Zahoora and A. S. Qureshi, “A Survey of the Recent Architectures of Deep Convolutional Neural Networks,” Artificial Intelligence Review, vol. 53, no. 8, pp. 5455–5516, 2020. doi · arXiv

  48. S. Khan, M. Naseer, M. Hayat, S. W. Zamir, F. S. Khan and M. Shah, “Transformers in Vision: A Survey,” ACM Computing Surveys, vol. 54, no. 10s, 2022. doi · arXiv

  49. S. Woo, S. Debnath, R. Hu, X. Chen, Z. Liu, I. S. Kweon and S. Xie, “ConvNeXt V2: Co-Designing and Scaling ConvNets With Masked Autoencoders,” in Proc. IEEE/CVF CVPR, 2023. CVF · arXiv

  50. M. Goldblum, H. Souri, R. Ni, M. Shu, V. Prabhu, G. Somepalli, P. Chattopadhyay, M. Ibrahim et al., “Battle of the Backbones: A Large-Scale Comparison of Pretrained Models across Computer Vision Tasks,” in Advances in Neural Information Processing Systems (NeurIPS), 2023. arXiv

  51. X. Chen, X. Wang, J. Zhou, Y. Qiao and C. Dong, “Activating More Pixels in Image Super-Resolution Transformer,” in Proc. IEEE/CVF CVPR, 2023. CVF · arXiv

  52. H. Guo, Y. Guo, Y. Zha, Y. Zhang, W. Li, T. Dai, S.-T. Xia and Y. Li, “MambaIRv2: Attentive State Space Restoration,” in Proc. IEEE/CVF CVPR, 2025. CVF · arXiv

  53. A. Araujo, W. Norris and J. Sim, “Computing Receptive Fields of Convolutional Neural Networks,” Distill, vol. 4, no. 11, 2019. Distill · doi

  54. X. Ding, X. Zhang, N. Ma, J. Han, G. Ding and J. Sun, “RepVGG: Making VGG-style ConvNets Great Again,” in Proc. IEEE/CVF CVPR, pp. 13733–13742, 2021. CVF paper

  55. S. Zagoruyko and N. Komodakis, “DiracNets: Training Very Deep Neural Networks Without Skip-Connections,” arXiv:1706.00388, 2017. arXiv

  56. X. Ding, Y. Guo, G. Ding and J. Han, “ACNet: Strengthening the Kernel Skeletons for Powerful CNN via Asymmetric Convolution Blocks,” in Proc. IEEE/CVF ICCV, 2019. CVF

  57. H. Zhao, J. Shi, X. Qi, X. Wang and J. Jia, “Pyramid Scene Parsing Network,” in Proc. IEEE CVPR, 2017. CVF

  58. M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth and B. Schiele, “The Cityscapes Dataset for Semantic Urban Scene Understanding,” in Proc. IEEE CVPR, 2016. CVF

  59. E. D. Cubuk, B. Zoph, D. Mane, V. Vasudevan and Q. V. Le, “AutoAugment: Learning Augmentation Strategies From Data,” in Proc. IEEE/CVF CVPR, 2019. CVF

  60. H. Zhang, M. Cisse, Y. N. Dauphin and D. Lopez-Paz, “mixup: Beyond Empirical Risk Minimization,” in Proc. ICLR, 2018. arXiv