批次正規化:在網路內部做標準化
選讀・DLCV 01・深度學習的影像處理與電腦視覺 的補充資料(第 1/2 則)。主線不依賴這則補充;引用編號對應本頁底部的參考文獻。
導覽:本頁與本系列: 導覽:本頁與本系列
想像每次考試都「看曲線」打分數:不管試題難易,分數都先平移、伸縮到全班平均為 0、典型離散程度為 1。批次正規化(batch normalization, BN)就是對一層輸出的每個通道(channel)做這件事,以目前這個小批次(mini-batch)內的影像當作「全班」,然後再讓網路用兩個學出來的數字把結果拉伸回想要的樣子 [37]。
取卷積層的某一個通道 。令 為這個通道在小批次的 張影像、所有 個位置上的全部數值,所以 (特徵圖上的每個位置因此都以同一種方式正規化 [37])。則
其中 與 是該通道在這個小批次中的平均與變異數, 是避免除以零的極小常數, 與 是該通道學出來的縮放與平移。如果網路想要回原本的數值,它可以學成 、,所以表達能力不會損失。在上面的殘差區塊裡,BN 位於卷積與 ReLU 之間;把最後一個 設為零( 維持初始值 0),分支輸出就是 ,這就是該節程式碼用的零初始化技巧。
訓練與推論的差別。 測試時,一張影像的輸出不應該取決於碰巧和它同批的其他影像。所以訓練時我們同時維護移動平均 (變異數同理), 為動量(momentum),下面的程式碼取 0.1。推論時用這些固定下來的數值取代 與 ,BN 就變成固定的逐通道線性映射,甚至可以併進前一層卷積的權重裡。原論文是對訓練期間的各個小批次取平均 [37];指數移動平均則是常見的實作方式。
程式碼 · Python · 44 行
import numpy as np
rng = np.random.default_rng(0)
C, eps, momentum = 3, 1e-5, 0.1
gamma, beta = np.ones(C), np.zeros(C) # learned scale and shift (one pair per channel)
run_mu, run_var = np.zeros(C), np.ones(C) # running statistics used at inference
def batch(n=32):
# feature maps (N, C, H, W) whose channels have very different statistics
scale = np.array([1.0, 5.0, 0.2]).reshape(1, C, 1, 1)
offset = np.array([0.0, 10.0, -3.0]).reshape(1, C, 1, 1)
return rng.normal(size=(n, C, 8, 8)) * scale + offset
def bn_train(x):
global run_mu, run_var
mu, var = x.mean(axis=(0, 2, 3)), x.var(axis=(0, 2, 3)) # statistics of this mini-batch
run_mu = (1 - momentum) * run_mu + momentum * mu
run_var = (1 - momentum) * run_var + momentum * var
xhat = (x - mu.reshape(1, C, 1, 1)) / np.sqrt(var.reshape(1, C, 1, 1) + eps)
return gamma.reshape(1, C, 1, 1) * xhat + beta.reshape(1, C, 1, 1)
def bn_eval(x):
xhat = (x - run_mu.reshape(1, C, 1, 1)) / np.sqrt(run_var.reshape(1, C, 1, 1) + eps)
return gamma.reshape(1, C, 1, 1) * xhat + beta.reshape(1, C, 1, 1)
x = batch()
y = bn_train(x)
print("input mean per channel:", x.mean(axis=(0, 2, 3)).round(2))
print("output mean per channel:", y.mean(axis=(0, 2, 3)).round(3))
print("output std per channel:", y.std(axis=(0, 2, 3)).round(3))
for _ in range(100): # many training batches update the running statistics
bn_train(batch())
print("running mean:", run_mu.round(2), " running var:", run_var.round(3))
x_new = batch()
print("eval-mode output mean:", bn_eval(x_new).mean(axis=(0, 2, 3)).round(2),
" std:", bn_eval(x_new).std(axis=(0, 2, 3)).round(2))
one = x_new[:1] # a single image
print("single image, eval mode, depends on batch? ",
np.allclose(bn_eval(one), bn_eval(x_new)[:1]))
print("single image, batch statistics of itself: max |train - eval| =",
np.abs(bn_train(one) - bn_eval(one)).max().round(2))以上述種子執行,印出:
程式碼 · 7 行
input mean per channel: [-0.02 9.89 -2.99]
output mean per channel: [-0. 0. -0.]
output std per channel: [1. 1. 1.]
running mean: [ 0.01 10. -3. ] running var: [ 1.011 24.831 0.04 ]
eval-mode output mean: [ 0.02 -0. 0.02] std: [1. 0.99 1. ]
single image, eval mode, depends on batch? True
single image, batch statistics of itself: max |train - eval| = 0.34三個通道一開始的平均與離散程度差異很大,經過 BN 後平均為 0、標準差為 1。訓練 101 個批次後,移動平均已接近資料的生成參數(平均 0、10、-3;變異數 1、25、0.04)。在評估(eval)模式下,一張影像不論單獨輸入或放在批次中,輸出都相同(True 那一行);但若用影像自己的統計量去正規化,結果就會不同(最後一行)。
常見錯誤。 忘了把模型切到推論模式(PyTorch 的 model.eval()),讓層繼續使用批次統計量;用極小的批次訓練,使 與 雜訊很大、與移動平均差很多;以及在統計特性與訓練資料不同的資料上評估,這時儲存的移動平均就不再適用。ViT 等 Transformer 改用層正規化(layer normalization),它對單一 token 的各個特徵計算統計量,因此訓練與推論的行為相同。
為什麼有效? 原本的解釋是「降低內部共變量偏移(internal covariate shift)」,也就是訓練時各層輸入分布的漂移 [37]。Santurkar 等人 [38] 檢驗了這個說法,報告指出層輸入分布的穩定與 BN 的成功關係不大,BN 真正的作用是讓最佳化地形(optimization landscape)顯著更平滑,使梯度更可預測。他們也指出,當時 BN 為何有效的確切原因仍未被充分理解。BN 與 LayerNorm、RMSNorm 的並列比較見 ML 02。
參考文獻
- Shyandram, “深度學習的數位影像處理介紹,” Medium post, 2024; updated 2026. link
- R. C. Gonzalez and R. E. Woods, Digital Image Processing, 4th ed., Pearson, 2018. publisher page
- S. Theodoridis and K. Koutroumbas, Pattern Recognition, 4th ed., Academic Press, 2008. publisher page
- Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel, “Backpropagation applied to handwritten zip code recognition,” Neural Computation, vol. 1, no. 4, pp. 541–551, 1989. doi:10.1162/neco.1989.1.4.541
- Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proceedings of the IEEE, vol. 86, no. 11, pp. 2278–2324, 1998. doi:10.1109/5.726791
- O. Russakovsky, J. Deng, H. Su, et al., “ImageNet large scale visual recognition challenge,” International Journal of Computer Vision, vol. 115, no. 3, pp. 211–252, 2015. doi:10.1007/s11263-015-0816-y · arXiv:1409.0575
- A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classification with deep convolutional neural networks,” NeurIPS, 2012. proceedings
- K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” ICLR, 2015. arXiv:1409.1556
- C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” CVPR, 2015. doi:10.1109/CVPR.2015.7298594 · arXiv:1409.4842
- K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” CVPR, 2016. doi:10.1109/CVPR.2016.90 · arXiv:1512.03385
- A. Krizhevsky, “Learning multiple layers of features from tiny images,” technical report, University of Toronto, 2009 (the CIFAR-10/100 datasets). dataset page
- W. Luo, Y. Li, R. Urtasun, and R. Zemel, “Understanding the effective receptive field in deep convolutional neural networks,” NeurIPS, 2016. arXiv:1701.04128
- J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” CVPR, 2015. arXiv:1411.4038
- O. Ronneberger, P. Fischer, and T. Brox, “U-Net: Convolutional networks for biomedical image segmentation,” MICCAI, 2015. arXiv:1505.04597
- A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” NeurIPS, 2017. ML anthology
- A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” ICLR, 2021. arXiv:2010.11929
- Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin Transformer: Hierarchical vision Transformer using shifted windows,” ICCV, 2021. doi:10.1109/ICCV48922.2021.00986 · arXiv:2103.14030
- Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie, “A ConvNet for the 2020s,” CVPR, 2022, arXiv:2201.03545; and S. Woo, S. Debnath, R. Hu, X. Chen, Z. Liu, I. S. Kweon, and S. Xie, “ConvNeXt V2: Co-designing and scaling ConvNets with masked autoencoders,” CVPR, 2023, arXiv:2301.00808.
- M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin, “Emerging properties in self-supervised vision Transformers,” ICCV, 2021. doi:10.1109/ICCV48922.2021.00951 · arXiv:2104.14294
- K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick, “Masked autoencoders are scalable vision learners,” CVPR, 2022. doi:10.1109/CVPR52688.2022.01553 · arXiv:2111.06377
- M. Oquab, T. Darcet, T. Moutakanni, et al., “DINOv2: Learning robust visual features without supervision,” Transactions on Machine Learning Research, 2024, arXiv:2304.07193; and O. Siméoni, H. V. Vo, M. Seitzer, F. Baldassarre, M. Oquab, et al., “DINOv3,” arXiv preprint, 2025, arXiv:2508.10104.
- L. Zhu, B. Liao, Q. Zhang, X. Wang, W. Liu, and X. Wang, “Vision Mamba: Efficient visual representation learning with bidirectional state space model,” ICML, 2024, arXiv:2401.09417; and Y. Liu, Y. Tian, Y. Zhao, H. Yu, L. Xie, Y. Wang, Q. Ye, J. Jiao, and Y. Liu, “VMamba: Visual state space model,” NeurIPS, 2024, arXiv:2401.10166.
- S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards real-time object detection with region proposal networks,” NeurIPS, 2015, arXiv:1506.01497; and N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with Transformers,” ECCV, 2020, arXiv:2005.12872.
- A. Kirillov, E. Mintun, N. Ravi, et al., “Segment anything,” ICCV, 2023, arXiv:2304.02643; N. Ravi, V. Gabeur, Y.-T. Hu, R. Hu, et al., “SAM 2: Segment anything in images and videos,” ICLR, 2025, arXiv:2408.00714; and N. Carion, L. Gustafson, Y.-T. Hu, S. Debnath, et al., “SAM 3: Segment anything with concepts,” ICLR, 2026, arXiv:2511.16719.
- A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, et al., “Learning transferable visual models from natural language supervision,” ICML, 2021. arXiv:2103.00020
- H. Liu, C. Li, Q. Wu, and Y. J. Lee, “Visual instruction tuning,” NeurIPS, 2023. arXiv:2304.08485
- J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” NeurIPS, 2020. arXiv:2006.11239
- R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” CVPR, 2022, arXiv:2112.10752; and W. Peebles and S. Xie, “Scalable diffusion models with Transformers,” ICCV, 2023, arXiv:2212.09748.
- T. Brooks, A. Holynski, and A. A. Efros, “InstructPix2Pix: Learning to follow image editing instructions,” CVPR, 2023. arXiv:2211.09800
- Z. Li, F. Liu, W. Yang, S. Peng, and J. Zhou, “A survey of convolutional neural networks: Analysis, applications, and prospects,” IEEE Transactions on Neural Networks and Learning Systems, vol. 33, 2022. doi:10.1109/TNNLS.2021.3084827 · arXiv:2004.02806
- S. Khan, M. Naseer, M. Hayat, S. W. Zamir, F. S. Khan, and M. Shah, “Transformers in vision: A survey,” ACM Computing Surveys, vol. 54, no. 10s, 2022. doi:10.1145/3505244 · arXiv:2101.01169
- L. Jing and Y. Tian, “Self-supervised visual feature learning with deep neural networks: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 43, 2021. doi:10.1109/TPAMI.2020.2992393 · arXiv:1902.06162
- R. Balestriero, M. Ibrahim, V. Sobal, et al., “A cookbook of self-supervised learning,” arXiv preprint, 2023. arXiv:2304.12210
- M. Awais, M. Naseer, S. Khan, R. M. Anwer, H. Cholakkal, M. Shah, M.-H. Yang, and F. S. Khan, “Foundation models defining a new era in vision: A survey and outlook,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 47, 2025. doi:10.1109/TPAMI.2024.3506283 · arXiv:2307.13721
- J. Zhang, J. Huang, S. Jin, and S. Lu, “Vision-language models for vision tasks: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, 2024. doi:10.1109/TPAMI.2024.3369699 · arXiv:2304.00685
- scikit-learn developers, “sklearn.datasets.load_digits” (a copy of the test set of the UCI optical recognition of handwritten digits dataset). docs
- S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” ICML, PMLR 37, pp. 448–456, 2015. PMLR · arXiv:1502.03167
- S. Santurkar, D. Tsipras, A. Ilyas, and A. Madry, “How does batch normalization help optimization?” NeurIPS, vol. 31, 2018. proceedings · arXiv:1805.11604
- G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” NIPS 2014 Deep Learning Workshop, arXiv preprint, 2015. arXiv:1503.02531