完整搜尋頁面 →

批次正規化:在網路內部做標準化

選讀・DLCV 01・深度學習的影像處理與電腦視覺 的補充資料(第 1/2 則)。主線不依賴這則補充;引用編號對應本頁底部的參考文獻。

導覽:本頁與本系列: 導覽:本頁與本系列

本頁內容

補充資料 · 2

  1. 批次正規化:在網路內部做標準化
  2. 知識蒸餾:向老師模型的軟答案學習

論文導讀 · 1

  1. 現代觀點綜述導讀

電腦視覺深度學習

  1. 02 · 深度影像分類
  2. 03 · 物件偵測:從 R-CNN 到 DETR
  3. 09 · 深度影像復原:低階視覺
查看完整課程樹 →

想像每次考試都「看曲線」打分數:不管試題難易,分數都先平移、伸縮到全班平均為 0、典型離散程度為 1。批次正規化(batch normalization, BN)就是對一層輸出的每個通道(channel)做這件事,以目前這個小批次(mini-batch)內的影像當作「全班」,然後再讓網路用兩個學出來的數字把結果拉伸回想要的樣子 [37]。

取卷積層的某一個通道 kk。令 BB 為這個通道在小批次的 mm 張影像、所有 H×WH \times W 個位置上的全部數值,所以 ∣B∣=mHW|B| = m H W(特徵圖上的每個位置因此都以同一種方式正規化 [37])。則

μB=1∣B∣∑x∈Bx,σB2=1∣B∣∑x∈B(x−μB)2,y=γ x−μBσB2+ϵ+β,\mu_B = \frac{1}{|B|} \sum_{x \in B} x, \qquad \sigma_B^2 = \frac{1}{|B|} \sum_{x \in B} (x - \mu_B)^2, \qquad y = \gamma\, \frac{x - \mu_B}{\sqrt{\sigma_B^2 + \epsilon}} + \beta ,

其中 μB\mu_B 與 σB2\sigma_B^2 是該通道在這個小批次中的平均與變異數,ϵ\epsilon 是避免除以零的極小常數,γ\gamma 與 β\beta 是該通道學出來的縮放與平移。如果網路想要回原本的數值,它可以學成 γ=σB2+ϵ\gamma = \sqrt{\sigma_B^2 + \epsilon}、β=μB\beta = \mu_B,所以表達能力不會損失。在上面的殘差區塊裡,BN 位於卷積與 ReLU 之間;把最後一個 γ\gamma 設為零(β\beta 維持初始值 0),分支輸出就是 F(x)=0F(\mathbf{x}) = 0,這就是該節程式碼用的零初始化技巧。

訓練與推論的差別。 測試時,一張影像的輸出不應該取決於碰巧和它同批的其他影像。所以訓練時我們同時維護移動平均 μrun←(1−ρ) μrun+ρ μB\mu_{\text{run}} \leftarrow (1-\rho)\,\mu_{\text{run}} + \rho\,\mu_B(變異數同理),ρ\rho 為動量(momentum),下面的程式碼取 0.1。推論時用這些固定下來的數值取代 μB\mu_B 與 σB2\sigma_B^2,BN 就變成固定的逐通道線性映射,甚至可以併進前一層卷積的權重裡。原論文是對訓練期間的各個小批次取平均 [37];指數移動平均則是常見的實作方式。

程式碼 · Python · 44 行
import numpy as np

rng = np.random.default_rng(0)
C, eps, momentum = 3, 1e-5, 0.1
gamma, beta = np.ones(C), np.zeros(C)             # learned scale and shift (one pair per channel)
run_mu, run_var = np.zeros(C), np.ones(C)         # running statistics used at inference

def batch(n=32):
    # feature maps (N, C, H, W) whose channels have very different statistics
    scale = np.array([1.0, 5.0, 0.2]).reshape(1, C, 1, 1)
    offset = np.array([0.0, 10.0, -3.0]).reshape(1, C, 1, 1)
    return rng.normal(size=(n, C, 8, 8)) * scale + offset

def bn_train(x):
    global run_mu, run_var
    mu, var = x.mean(axis=(0, 2, 3)), x.var(axis=(0, 2, 3))     # statistics of this mini-batch
    run_mu = (1 - momentum) * run_mu + momentum * mu
    run_var = (1 - momentum) * run_var + momentum * var
    xhat = (x - mu.reshape(1, C, 1, 1)) / np.sqrt(var.reshape(1, C, 1, 1) + eps)
    return gamma.reshape(1, C, 1, 1) * xhat + beta.reshape(1, C, 1, 1)

def bn_eval(x):
    xhat = (x - run_mu.reshape(1, C, 1, 1)) / np.sqrt(run_var.reshape(1, C, 1, 1) + eps)
    return gamma.reshape(1, C, 1, 1) * xhat + beta.reshape(1, C, 1, 1)

x = batch()
y = bn_train(x)
print("input  mean per channel:", x.mean(axis=(0, 2, 3)).round(2))
print("output mean per channel:", y.mean(axis=(0, 2, 3)).round(3))
print("output std  per channel:", y.std(axis=(0, 2, 3)).round(3))

for _ in range(100):                               # many training batches update the running statistics
    bn_train(batch())
print("running mean:", run_mu.round(2), " running var:", run_var.round(3))

x_new = batch()
print("eval-mode output mean:", bn_eval(x_new).mean(axis=(0, 2, 3)).round(2),
      " std:", bn_eval(x_new).std(axis=(0, 2, 3)).round(2))

one = x_new[:1]                                    # a single image
print("single image, eval mode, depends on batch? ",
      np.allclose(bn_eval(one), bn_eval(x_new)[:1]))
print("single image, batch statistics of itself: max |train - eval| =",
      np.abs(bn_train(one) - bn_eval(one)).max().round(2))

以上述種子執行,印出:

程式碼 · 7 行
input  mean per channel: [-0.02  9.89 -2.99]
output mean per channel: [-0.  0. -0.]
output std  per channel: [1. 1. 1.]
running mean: [ 0.01 10.   -3.  ]  running var: [ 1.011 24.831  0.04 ]
eval-mode output mean: [ 0.02 -0.    0.02]  std: [1.   0.99 1.  ]
single image, eval mode, depends on batch?  True
single image, batch statistics of itself: max |train - eval| = 0.34

三個通道一開始的平均與離散程度差異很大,經過 BN 後平均為 0、標準差為 1。訓練 101 個批次後,移動平均已接近資料的生成參數(平均 0、10、-3;變異數 1、25、0.04)。在評估(eval)模式下,一張影像不論單獨輸入或放在批次中,輸出都相同(True 那一行);但若用影像自己的統計量去正規化,結果就會不同(最後一行)。

常見錯誤。 忘了把模型切到推論模式(PyTorch 的 model.eval()),讓層繼續使用批次統計量;用極小的批次訓練,使 μB\mu_B 與 σB2\sigma_B^2 雜訊很大、與移動平均差很多;以及在統計特性與訓練資料不同的資料上評估,這時儲存的移動平均就不再適用。ViT 等 Transformer 改用層正規化(layer normalization),它對單一 token 的各個特徵計算統計量,因此訓練與推論的行為相同。

為什麼有效? 原本的解釋是「降低內部共變量偏移(internal covariate shift)」,也就是訓練時各層輸入分布的漂移 [37]。Santurkar 等人 [38] 檢驗了這個說法,報告指出層輸入分布的穩定與 BN 的成功關係不大,BN 真正的作用是讓最佳化地形(optimization landscape)顯著更平滑,使梯度更可預測。他們也指出,當時 BN 為何有效的確切原因仍未被充分理解。BN 與 LayerNorm、RMSNorm 的並列比較見 ML 02。

參考文獻

  1. Shyandram, “深度學習的數位影像處理介紹,” Medium post, 2024; updated 2026. link
  2. R. C. Gonzalez and R. E. Woods, Digital Image Processing, 4th ed., Pearson, 2018. publisher page
  3. S. Theodoridis and K. Koutroumbas, Pattern Recognition, 4th ed., Academic Press, 2008. publisher page
  4. Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel, “Backpropagation applied to handwritten zip code recognition,” Neural Computation, vol. 1, no. 4, pp. 541–551, 1989. doi:10.1162/neco.1989.1.4.541
  5. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proceedings of the IEEE, vol. 86, no. 11, pp. 2278–2324, 1998. doi:10.1109/5.726791
  6. O. Russakovsky, J. Deng, H. Su, et al., “ImageNet large scale visual recognition challenge,” International Journal of Computer Vision, vol. 115, no. 3, pp. 211–252, 2015. doi:10.1007/s11263-015-0816-y · arXiv:1409.0575
  7. A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classification with deep convolutional neural networks,” NeurIPS, 2012. proceedings
  8. K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” ICLR, 2015. arXiv:1409.1556
  9. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” CVPR, 2015. doi:10.1109/CVPR.2015.7298594 · arXiv:1409.4842
  10. K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” CVPR, 2016. doi:10.1109/CVPR.2016.90 · arXiv:1512.03385
  11. A. Krizhevsky, “Learning multiple layers of features from tiny images,” technical report, University of Toronto, 2009 (the CIFAR-10/100 datasets). dataset page
  12. W. Luo, Y. Li, R. Urtasun, and R. Zemel, “Understanding the effective receptive field in deep convolutional neural networks,” NeurIPS, 2016. arXiv:1701.04128
  13. J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” CVPR, 2015. arXiv:1411.4038
  14. O. Ronneberger, P. Fischer, and T. Brox, “U-Net: Convolutional networks for biomedical image segmentation,” MICCAI, 2015. arXiv:1505.04597
  15. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” NeurIPS, 2017. ML anthology
  16. A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” ICLR, 2021. arXiv:2010.11929
  17. Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin Transformer: Hierarchical vision Transformer using shifted windows,” ICCV, 2021. doi:10.1109/ICCV48922.2021.00986 · arXiv:2103.14030
  18. Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie, “A ConvNet for the 2020s,” CVPR, 2022, arXiv:2201.03545; and S. Woo, S. Debnath, R. Hu, X. Chen, Z. Liu, I. S. Kweon, and S. Xie, “ConvNeXt V2: Co-designing and scaling ConvNets with masked autoencoders,” CVPR, 2023, arXiv:2301.00808.
  19. M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin, “Emerging properties in self-supervised vision Transformers,” ICCV, 2021. doi:10.1109/ICCV48922.2021.00951 · arXiv:2104.14294
  20. K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick, “Masked autoencoders are scalable vision learners,” CVPR, 2022. doi:10.1109/CVPR52688.2022.01553 · arXiv:2111.06377
  21. M. Oquab, T. Darcet, T. Moutakanni, et al., “DINOv2: Learning robust visual features without supervision,” Transactions on Machine Learning Research, 2024, arXiv:2304.07193; and O. Siméoni, H. V. Vo, M. Seitzer, F. Baldassarre, M. Oquab, et al., “DINOv3,” arXiv preprint, 2025, arXiv:2508.10104.
  22. L. Zhu, B. Liao, Q. Zhang, X. Wang, W. Liu, and X. Wang, “Vision Mamba: Efficient visual representation learning with bidirectional state space model,” ICML, 2024, arXiv:2401.09417; and Y. Liu, Y. Tian, Y. Zhao, H. Yu, L. Xie, Y. Wang, Q. Ye, J. Jiao, and Y. Liu, “VMamba: Visual state space model,” NeurIPS, 2024, arXiv:2401.10166.
  23. S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards real-time object detection with region proposal networks,” NeurIPS, 2015, arXiv:1506.01497; and N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with Transformers,” ECCV, 2020, arXiv:2005.12872.
  24. A. Kirillov, E. Mintun, N. Ravi, et al., “Segment anything,” ICCV, 2023, arXiv:2304.02643; N. Ravi, V. Gabeur, Y.-T. Hu, R. Hu, et al., “SAM 2: Segment anything in images and videos,” ICLR, 2025, arXiv:2408.00714; and N. Carion, L. Gustafson, Y.-T. Hu, S. Debnath, et al., “SAM 3: Segment anything with concepts,” ICLR, 2026, arXiv:2511.16719.
  25. A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, et al., “Learning transferable visual models from natural language supervision,” ICML, 2021. arXiv:2103.00020
  26. H. Liu, C. Li, Q. Wu, and Y. J. Lee, “Visual instruction tuning,” NeurIPS, 2023. arXiv:2304.08485
  27. J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” NeurIPS, 2020. arXiv:2006.11239
  28. R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” CVPR, 2022, arXiv:2112.10752; and W. Peebles and S. Xie, “Scalable diffusion models with Transformers,” ICCV, 2023, arXiv:2212.09748.
  29. T. Brooks, A. Holynski, and A. A. Efros, “InstructPix2Pix: Learning to follow image editing instructions,” CVPR, 2023. arXiv:2211.09800
  30. Z. Li, F. Liu, W. Yang, S. Peng, and J. Zhou, “A survey of convolutional neural networks: Analysis, applications, and prospects,” IEEE Transactions on Neural Networks and Learning Systems, vol. 33, 2022. doi:10.1109/TNNLS.2021.3084827 · arXiv:2004.02806
  31. S. Khan, M. Naseer, M. Hayat, S. W. Zamir, F. S. Khan, and M. Shah, “Transformers in vision: A survey,” ACM Computing Surveys, vol. 54, no. 10s, 2022. doi:10.1145/3505244 · arXiv:2101.01169
  32. L. Jing and Y. Tian, “Self-supervised visual feature learning with deep neural networks: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 43, 2021. doi:10.1109/TPAMI.2020.2992393 · arXiv:1902.06162
  33. R. Balestriero, M. Ibrahim, V. Sobal, et al., “A cookbook of self-supervised learning,” arXiv preprint, 2023. arXiv:2304.12210
  34. M. Awais, M. Naseer, S. Khan, R. M. Anwer, H. Cholakkal, M. Shah, M.-H. Yang, and F. S. Khan, “Foundation models defining a new era in vision: A survey and outlook,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 47, 2025. doi:10.1109/TPAMI.2024.3506283 · arXiv:2307.13721
  35. J. Zhang, J. Huang, S. Jin, and S. Lu, “Vision-language models for vision tasks: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, 2024. doi:10.1109/TPAMI.2024.3369699 · arXiv:2304.00685
  36. scikit-learn developers, “sklearn.datasets.load_digits” (a copy of the test set of the UCI optical recognition of handwritten digits dataset). docs
  37. S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” ICML, PMLR 37, pp. 448–456, 2015. PMLR · arXiv:1502.03167
  38. S. Santurkar, D. Tsipras, A. Ilyas, and A. Madry, “How does batch normalization help optimization?” NeurIPS, vol. 31, 2018. proceedings · arXiv:1805.11604
  39. G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” NIPS 2014 Deep Learning Workshop, arXiv preprint, 2015. arXiv:1503.02531