Full search page →

Batch normalization: standardize inside the network

Optional · supplementary note 1 of 2 for DLCV 01 · Deep Learning for Image Processing and Computer Vision. Nothing in the main text depends on it; citation numbers refer to the reference list at the bottom of this page.

Navigate: this page and this track: Navigate: this page and this track

On this page

Supplementary notes · 2

  1. Batch normalization: standardize inside the network
  2. Knowledge distillation: learning from a teacher’s soft answers

Paper guides · 1

  1. Guide to the Modern view surveys

Deep Learning for Computer Vision

  1. 02 · Deep Image Classification
View the whole course tree →

Imagine grading every exam on a curve: whatever the difficulty, scores are shifted and stretched so the class average is 0 and the typical spread is 1. Batch normalization (BN) does this to every channel of a layer’s output, using the images in the current mini-batch as “the class”, and then lets the network re-stretch the result with two learned numbers [37].

Take one channel kk of a convolutional layer. Let BB be the set of all values of that channel over the mm images of the mini-batch and over all H×WH \times W positions, so ∣B∣=mHW|B| = m H W (every position of a feature map is thus normalized in the same way [37]). Then

μB=1∣B∣∑x∈Bx,σB2=1∣B∣∑x∈B(x−μB)2,y=γ x−μBσB2+ϵ+β,\mu_B = \frac{1}{|B|} \sum_{x \in B} x, \qquad \sigma_B^2 = \frac{1}{|B|} \sum_{x \in B} (x - \mu_B)^2, \qquad y = \gamma\, \frac{x - \mu_B}{\sqrt{\sigma_B^2 + \epsilon}} + \beta ,

where μB\mu_B and σB2\sigma_B^2 are the mean and variance of the channel in this mini-batch, ϵ\epsilon is a tiny constant that avoids division by zero, and γ\gamma and β\beta are a learned scale and shift for the channel. If the network wants the original values back, it can learn γ=σB2+ϵ\gamma = \sqrt{\sigma_B^2 + \epsilon} and β=μB\beta = \mu_B, so no capacity is lost. In the residual block above, BN sits between the convolution and the ReLU; setting the last γ\gamma to zero (with β\beta at its initial value 0) makes the branch output F(x)=0F(\mathbf{x}) = 0, which is the zero-initialization trick used in the code of that section.

Training versus inference. At test time one image should not depend on whichever other images happen to share its batch. So during training we also keep running averages, μrun←(1−ρ) μrun+ρ μB\mu_{\text{run}} \leftarrow (1-\rho)\,\mu_{\text{run}} + \rho\,\mu_B (the same for the variance), with momentum ρ\rho (0.1 in the code below). At inference these frozen values replace μB\mu_B and σB2\sigma_B^2, so BN becomes a fixed per-channel linear map, which can even be folded into the preceding convolution. The original paper averages the statistics over training mini-batches [37]; the exponential moving average is the usual implementation.

Code · Python · 44 lines
import numpy as np

rng = np.random.default_rng(0)
C, eps, momentum = 3, 1e-5, 0.1
gamma, beta = np.ones(C), np.zeros(C)             # learned scale and shift (one pair per channel)
run_mu, run_var = np.zeros(C), np.ones(C)         # running statistics used at inference

def batch(n=32):
    # feature maps (N, C, H, W) whose channels have very different statistics
    scale = np.array([1.0, 5.0, 0.2]).reshape(1, C, 1, 1)
    offset = np.array([0.0, 10.0, -3.0]).reshape(1, C, 1, 1)
    return rng.normal(size=(n, C, 8, 8)) * scale + offset

def bn_train(x):
    global run_mu, run_var
    mu, var = x.mean(axis=(0, 2, 3)), x.var(axis=(0, 2, 3))     # statistics of this mini-batch
    run_mu = (1 - momentum) * run_mu + momentum * mu
    run_var = (1 - momentum) * run_var + momentum * var
    xhat = (x - mu.reshape(1, C, 1, 1)) / np.sqrt(var.reshape(1, C, 1, 1) + eps)
    return gamma.reshape(1, C, 1, 1) * xhat + beta.reshape(1, C, 1, 1)

def bn_eval(x):
    xhat = (x - run_mu.reshape(1, C, 1, 1)) / np.sqrt(run_var.reshape(1, C, 1, 1) + eps)
    return gamma.reshape(1, C, 1, 1) * xhat + beta.reshape(1, C, 1, 1)

x = batch()
y = bn_train(x)
print("input  mean per channel:", x.mean(axis=(0, 2, 3)).round(2))
print("output mean per channel:", y.mean(axis=(0, 2, 3)).round(3))
print("output std  per channel:", y.std(axis=(0, 2, 3)).round(3))

for _ in range(100):                               # many training batches update the running statistics
    bn_train(batch())
print("running mean:", run_mu.round(2), " running var:", run_var.round(3))

x_new = batch()
print("eval-mode output mean:", bn_eval(x_new).mean(axis=(0, 2, 3)).round(2),
      " std:", bn_eval(x_new).std(axis=(0, 2, 3)).round(2))

one = x_new[:1]                                    # a single image
print("single image, eval mode, depends on batch? ",
      np.allclose(bn_eval(one), bn_eval(x_new)[:1]))
print("single image, batch statistics of itself: max |train - eval| =",
      np.abs(bn_train(one) - bn_eval(one)).max().round(2))

Run with the seed shown, this prints:

Code · 7 lines
input  mean per channel: [-0.02  9.89 -2.99]
output mean per channel: [-0.  0. -0.]
output std  per channel: [1. 1. 1.]
running mean: [ 0.01 10.   -3.  ]  running var: [ 1.011 24.831  0.04 ]
eval-mode output mean: [ 0.02 -0.    0.02]  std: [1.   0.99 1.  ]
single image, eval mode, depends on batch?  True
single image, batch statistics of itself: max |train - eval| = 0.34

The three channels start with very different means and spreads and leave BN with mean 0 and standard deviation 1. After 101 training batches the running statistics are close to the values the data was generated with (means 0, 10, -3; variances 1, 25, 0.04). In eval mode an image gets the same output whether it is alone or inside a batch (the True line), whereas normalizing one image by its own statistics would give a different result (the last line).

Common mistakes. Forgetting to switch the model to inference mode (model.eval() in PyTorch), so the layer keeps using batch statistics; training with very small batches, where μB\mu_B and σB2\sigma_B^2 are noisy and differ from the running values; and evaluating on data whose statistics differ from the training data, where the stored running values no longer fit. Transformers such as ViT use layer normalization instead, which computes the statistics over the features of one token and therefore behaves the same in training and inference.

Why does it help? The original explanation was “reducing internal covariate shift”, the drift of each layer’s input distribution during training [37]. Santurkar et al. [38] tested this and reported that distributional stability of layer inputs has little to do with BN’s success, and that BN instead makes the optimization landscape smoother, giving more predictive gradients. They note that the exact reasons for BN’s effectiveness were still poorly understood. See ML 02 for BN next to LayerNorm and RMSNorm.

References

  1. Shyandram, “深度學習的數位影像處理介紹 (An introduction to digital image processing with deep learning),” Medium post, 2024; updated 2026. link
  2. R. C. Gonzalez and R. E. Woods, Digital Image Processing, 4th ed., Pearson, 2018. publisher page
  3. S. Theodoridis and K. Koutroumbas, Pattern Recognition, 4th ed., Academic Press, 2008. publisher page
  4. Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel, “Backpropagation applied to handwritten zip code recognition,” Neural Computation, vol. 1, no. 4, pp. 541–551, 1989. doi:10.1162/neco.1989.1.4.541
  5. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proceedings of the IEEE, vol. 86, no. 11, pp. 2278–2324, 1998. doi:10.1109/5.726791
  6. O. Russakovsky, J. Deng, H. Su, et al., “ImageNet large scale visual recognition challenge,” International Journal of Computer Vision, vol. 115, no. 3, pp. 211–252, 2015. doi:10.1007/s11263-015-0816-y · arXiv:1409.0575
  7. A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classification with deep convolutional neural networks,” NeurIPS, 2012. proceedings
  8. K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” ICLR, 2015. arXiv:1409.1556
  9. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” CVPR, 2015. doi:10.1109/CVPR.2015.7298594 · arXiv:1409.4842
  10. K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” CVPR, 2016. doi:10.1109/CVPR.2016.90 · arXiv:1512.03385
  11. A. Krizhevsky, “Learning multiple layers of features from tiny images,” technical report, University of Toronto, 2009 (the CIFAR-10/100 datasets). dataset page
  12. W. Luo, Y. Li, R. Urtasun, and R. Zemel, “Understanding the effective receptive field in deep convolutional neural networks,” NeurIPS, 2016. arXiv:1701.04128
  13. J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” CVPR, 2015. arXiv:1411.4038
  14. O. Ronneberger, P. Fischer, and T. Brox, “U-Net: Convolutional networks for biomedical image segmentation,” MICCAI, 2015. arXiv:1505.04597
  15. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” NeurIPS, 2017. ML anthology
  16. A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” ICLR, 2021. arXiv:2010.11929
  17. Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin Transformer: Hierarchical vision Transformer using shifted windows,” ICCV, 2021. doi:10.1109/ICCV48922.2021.00986 · arXiv:2103.14030
  18. Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie, “A ConvNet for the 2020s,” CVPR, 2022, arXiv:2201.03545; and S. Woo, S. Debnath, R. Hu, X. Chen, Z. Liu, I. S. Kweon, and S. Xie, “ConvNeXt V2: Co-designing and scaling ConvNets with masked autoencoders,” CVPR, 2023, arXiv:2301.00808.
  19. M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin, “Emerging properties in self-supervised vision Transformers,” ICCV, 2021. doi:10.1109/ICCV48922.2021.00951 · arXiv:2104.14294
  20. K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick, “Masked autoencoders are scalable vision learners,” CVPR, 2022. doi:10.1109/CVPR52688.2022.01553 · arXiv:2111.06377
  21. M. Oquab, T. Darcet, T. Moutakanni, et al., “DINOv2: Learning robust visual features without supervision,” Transactions on Machine Learning Research, 2024, arXiv:2304.07193; and O. Siméoni, H. V. Vo, M. Seitzer, F. Baldassarre, M. Oquab, et al., “DINOv3,” arXiv preprint, 2025, arXiv:2508.10104.
  22. L. Zhu, B. Liao, Q. Zhang, X. Wang, W. Liu, and X. Wang, “Vision Mamba: Efficient visual representation learning with bidirectional state space model,” ICML, 2024, arXiv:2401.09417; and Y. Liu, Y. Tian, Y. Zhao, H. Yu, L. Xie, Y. Wang, Q. Ye, J. Jiao, and Y. Liu, “VMamba: Visual state space model,” NeurIPS, 2024, arXiv:2401.10166.
  23. S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards real-time object detection with region proposal networks,” NeurIPS, 2015, arXiv:1506.01497; and N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with Transformers,” ECCV, 2020, arXiv:2005.12872.
  24. A. Kirillov, E. Mintun, N. Ravi, et al., “Segment anything,” ICCV, 2023, arXiv:2304.02643; N. Ravi, V. Gabeur, Y.-T. Hu, R. Hu, et al., “SAM 2: Segment anything in images and videos,” ICLR, 2025, arXiv:2408.00714; and N. Carion, L. Gustafson, Y.-T. Hu, S. Debnath, et al., “SAM 3: Segment anything with concepts,” ICLR, 2026, arXiv:2511.16719.
  25. A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, et al., “Learning transferable visual models from natural language supervision,” ICML, 2021. arXiv:2103.00020
  26. H. Liu, C. Li, Q. Wu, and Y. J. Lee, “Visual instruction tuning,” NeurIPS, 2023. arXiv:2304.08485
  27. J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” NeurIPS, 2020. arXiv:2006.11239
  28. R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” CVPR, 2022, arXiv:2112.10752; and W. Peebles and S. Xie, “Scalable diffusion models with Transformers,” ICCV, 2023, arXiv:2212.09748.
  29. T. Brooks, A. Holynski, and A. A. Efros, “InstructPix2Pix: Learning to follow image editing instructions,” CVPR, 2023. arXiv:2211.09800
  30. Z. Li, F. Liu, W. Yang, S. Peng, and J. Zhou, “A survey of convolutional neural networks: Analysis, applications, and prospects,” IEEE Transactions on Neural Networks and Learning Systems, vol. 33, 2022. doi:10.1109/TNNLS.2021.3084827 · arXiv:2004.02806
  31. S. Khan, M. Naseer, M. Hayat, S. W. Zamir, F. S. Khan, and M. Shah, “Transformers in vision: A survey,” ACM Computing Surveys, vol. 54, no. 10s, 2022. doi:10.1145/3505244 · arXiv:2101.01169
  32. L. Jing and Y. Tian, “Self-supervised visual feature learning with deep neural networks: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 43, 2021. doi:10.1109/TPAMI.2020.2992393 · arXiv:1902.06162
  33. R. Balestriero, M. Ibrahim, V. Sobal, et al., “A cookbook of self-supervised learning,” arXiv preprint, 2023. arXiv:2304.12210
  34. M. Awais, M. Naseer, S. Khan, R. M. Anwer, H. Cholakkal, M. Shah, M.-H. Yang, and F. S. Khan, “Foundation models defining a new era in vision: A survey and outlook,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 47, 2025. doi:10.1109/TPAMI.2024.3506283 · arXiv:2307.13721
  35. J. Zhang, J. Huang, S. Jin, and S. Lu, “Vision-language models for vision tasks: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, 2024. doi:10.1109/TPAMI.2024.3369699 · arXiv:2304.00685
  36. scikit-learn developers, “sklearn.datasets.load_digits” (a copy of the test set of the UCI optical recognition of handwritten digits dataset). docs
  37. S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” ICML, PMLR 37, pp. 448–456, 2015. PMLR · arXiv:1502.03167
  38. S. Santurkar, D. Tsipras, A. Ilyas, and A. Madry, “How does batch normalization help optimization?” NeurIPS, vol. 31, 2018. proceedings · arXiv:1805.11604
  39. G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” NIPS 2014 Deep Learning Workshop, arXiv preprint, 2015. arXiv:1503.02531