RepVGG: training with branches, deploying a plain network
Optional · paper guide 4 of 4 for DLCV 01B · Building Blocks and How to Compose Them: Classification and Restoration Networks. Citation numbers refer to the reference list at the bottom of this page.
Navigate: this page and this track: Navigate: this page and this track
RepVGG [54] separates the network that is convenient to train from the network that is convenient to run. Its contribution is an exact conversion of trained blocks, rather than a claim that one convolution can reproduce arbitrary nonlinear branches.
Challenge
A deployment engine can execute a sequence of ordinary convolutions efficiently, yet that simple inference graph is not necessarily the easiest graph to optimize during training. A residual branch supplies an additional route for information and gradients; it also requires intermediate activations and an addition when the model runs. RepVGG asks whether those training benefits can be retained without retaining the branches at inference.
The paper targets throughput on GPUs and possible specialized hardware. It emphasizes that operation counts alone do not describe memory access, kernel support or parallel execution. Its argument is therefore about a measured deployment trade-off, not about making every model as small as possible. In particular, it does not establish that a dense convolutional network is preferable on every mobile processor.
Method

Figure source: [54], Published paper PDF, PDF page 5. Original PDF.
The training block. A block adds three branches: a convolution followed by batch normalization (BN), a convolution followed by BN, and an identity followed by BN. One ReLU follows their sum. The identity branch exists only when the spatial stride is one and the input and output channel counts agree. Downsampling blocks use the two convolutional branches. The convolutions use compatible stride, padding and grouping so that their outputs can be added.
That placement of the nonlinearity is essential. BN with frozen inference statistics is an affine transform, so all three branches can become affine convolutions. A ReLU inside each branch would prevent their replacement by one linear convolution followed by one ReLU. During training, BN uses batch statistics; the conversion is not an equivalence between the training dynamics of a branched block and those of a plain convolution.
Fold each convolution–BN pair. Write a branch’s convolution as , where is the input, indexes output channels, is that output channel’s kernel, denotes convolution and is its optional bias. Let and be BN’s stored mean and variance, and its learned scale and shift, and its stability constant. Evaluation-mode BN is absorbed into
Here scales every weight of output channel ; and are the converted kernel and bias. A convolution without a bias has . These are stored-statistic calculations, with no calibration batch needed for the algebra itself.
Put the branches in the same kernel space. Place the folded kernel at the centre of a zero-filled kernel. For a dense identity branch, construct a kernel with a one at the centre for matching input/output channels and zeros elsewhere, then fold its BN with the same formula. Grouped blocks require the corresponding identity within each group; the identity branch cannot mix groups. The paper uses the same group count in the two convolutional branches.
If centre-pads a kernel, and are the folded convolution kernels, and is the folded identity kernel, the deployed parameters are
A missing identity contributes zero kernel and bias. and define one convolution; the original post-sum ReLU remains after it. This conversion removes the branches and their BN operators from the inference graph. It is not an approximation or a further learning stage. Floating-point evaluation can differ slightly because the order of arithmetic changes.
The step-by-step folding note already checks a complete three-branch block. This smaller check isolates a detail that is easy to get wrong: an identity with BN becomes a channel-scaled centre kernel and a bias, not just an identity kernel.
Code · Python · 24 lines
import numpy as np
from numpy.lib.stride_tricks import sliding_window_view
rng = np.random.default_rng(4)
x = rng.normal(size=(1, 2, 5, 5)) # batch, channels, height, width
gamma = np.array([1.2, -0.7])
beta = np.array([0.1, 0.3])
mean = np.array([0.2, -0.4])
variance = np.array([0.5, 1.3])
scale = gamma / np.sqrt(variance + 1e-5)
kernel = np.zeros((2, 2, 3, 3)) # output, input, kernel height/width
kernel[np.arange(2), np.arange(2), 1, 1] = scale
bias = beta - scale * mean
windows = sliding_window_view(
np.pad(x, ((0, 0), (0, 0), (1, 1), (1, 1))), (3, 3), axis=(2, 3)
)
fused = np.einsum("nchwkl,ockl->nohw", windows, kernel) + bias[None, :, None, None]
branch = (
gamma[None, :, None, None] * (x - mean[None, :, None, None])
/ np.sqrt(variance[None, :, None, None] + 1e-5)
+ beta[None, :, None, None]
)
print(f"max error: {np.max(np.abs(fused - branch)):.3e}")
assert np.allclose(fused, branch, rtol=1e-12, atol=1e-12)The tested output is max error: 4.441e-16. This is an arithmetic check with synthetic inputs, not a trained-model accuracy experiment.
The whole architecture. The body has five stages, each beginning with a stride-two convolution; global average pooling and a classifier follow it. Table 2 gives stage block counts of for the A family and for B. For width multipliers and , the stage widths are , , , and . The first stage is capped rather than widened to avoid expensive processing of large feature maps. A separate multiplier for the final stage permits a wider low-resolution representation.
Table 3 sets to , and for A0/A1/A2, and , , and for B0/B1/B2/B3. The g2/g4 variants use two/four groups in selected alternating convolution layers, with dense convolutions between them to allow inter-channel exchange. They are deployment variants with a different accuracy/cost balance, not a different folding principle.
Training and timing. The simple ImageNet recipe uses random crops and horizontal flips, SGD, a global batch of 256, initial learning rate 0.1, cosine decay over 120 epochs, momentum 0.9 and weight decay on convolution/classifier kernels. The stronger 200-epoch recipe adds five warmup epochs, AutoAugment [59], label smoothing and mixup [60]. Throughput uses one GTX 1080 Ti, FP32 and batch size 128, after 50 warmup batches followed by 50 timed batches. Baseline convolution–BN sequences are fused too. These settings matter when interpreting the results.
Results
Tables 4 and 5 report the following ImageNet top-1 accuracy and throughput. The rows retain their training recipe; a higher score after a stronger recipe is not an architecture-only improvement.
| Model | Training epochs | Top-1 (%) | Images/s |
|---|---|---|---|
| RepVGG-A0 | 120 | 72.41 | 3256 |
| ResNet-18 | 120 | 71.16 | 2442 |
| RepVGG-A2 | 120 | 76.48 | 1322 |
| ResNet-50 | 120 | 76.31 | 719 |
| RepVGG-B1g4 | 120 | 77.58 | 868 |
| ResNet-101 | 120 | 77.21 | 430 |
| RepVGG-B3 | 200 | 80.52 | 363 |
| RegNetX-12GF | 200 | 80.55 | 277 |
All rows shown use ImageNet inputs. Those are throughput measurements under the paper’s batch-128 protocol, not batch-one latency or a guarantee for another device. In the Cityscapes [58] validation experiment, Table 8 reports a PSPNet [57] with RepVGG-B1g2-fast at 78.88% mIoU and 10.9 images/s, versus the ResNet-50 backbone at 77.17% and 10.4 images/s. This timing instead uses FP32, batch 16 and inputs on the same GPU. The “fast” variant limits dilation to the final five layers; the comparison therefore includes a concrete dilation choice, rather than showing that all deployment operators have equal efficiency.
Ablations
Branch removal, Table 6. With the same final B0 inference structure, training without either auxiliary branch reaches 72.39% top-1. Adding only identity–BN gives 74.79%; adding only –BN gives 73.15%; using both gives 75.14%. The full block runs at 1061 images/s before conversion, versus the deployed B0’s 1817 in Table 4. Thus the extra training graph helps optimization, and removing it after training recovers throughput; the final architecture alone does not explain the accuracy difference.
Placement and nonlinearities, Table 7. Removing BN from the identity branch gives 74.18%; moving BN after the branch sum gives 73.52%, against 75.14% for the proposed block. Adding a ReLU inside each branch reaches 75.69%, but destroys the one-convolution conversion. This is an instructive trade-off: an accuracy gain is not a valid replacement when it breaks the deployment constraint.
Alternative parameterizations, Table 7 and Section 4.2. The paper’s B0 comparisons report 73.97% for DiracNet-style parameterization [55], 73.51% for directly adding an identity kernel, 73.58% for the asymmetric convolution block [56] and 74.56% for reorganizing the layers into residual blocks. These comparisons concern the variants trained in this experiment; they do not rank every possible training recipe for those methods. Replacing ResNet-50 convolutions with RepVGG blocks gives 76.34%, versus the 76.31% baseline. The nearly unchanged result is evidence against treating the block as a universal accuracy upgrade for an already residual network.
Conclusion
The useful design pattern is to choose a trainable graph whose linear inference branches can be converted exactly into a simpler graph. RepVGG demonstrates that pattern for plain convolutional classifiers and segmentation backbones. The proof of equivalence, the accuracy comparison and the hardware timing answer three different questions; a sound deployment decision needs all three.
Problems
The paper explicitly prioritizes GPU throughput over the smallest parameter count. Its results do not settle low-power or batch-one deployment, and the training branches still require memory and computation. Folding relies on frozen, meaningful BN statistics; it does not repair a poorly calibrated model. Subsequent quantization can introduce rounding error, so floating-point folding equivalence is not a proof of integer-inference equivalence. See DLCV 01C for the separate quantization and deployment questions, and DLCV 03A for detector designs where training and inference structures can differ. Neither the paper’s accuracy nor its images/s should be carried into those tasks without a new evaluation.
References
-
K. He, X. Zhang, S. Ren and J. Sun, “Deep Residual Learning for Image Recognition,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. CVF · arXiv
-
Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell and S. Xie, “A ConvNet for the 2020s,” in Proc. IEEE/CVF CVPR, 2022. CVF · arXiv
-
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer et al., “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale,” in International Conference on Learning Representations (ICLR), 2021. arXiv
-
J. Liang, J. Cao, G. Sun, K. Zhang, L. Van Gool and R. Timofte, “SwinIR: Image Restoration Using Swin Transformer,” in Proc. IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), 2021. CVF · arXiv
-
T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan and S. Belongie, “Feature Pyramid Networks for Object Detection,” in Proc. IEEE CVPR, 2017. CVF · arXiv
-
Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin and B. Guo, “Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows,” in Proc. IEEE/CVF International Conference on Computer Vision (ICCV), 2021. CVF · arXiv
-
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy et al., “ImageNet Large Scale Visual Recognition Challenge,” International Journal of Computer Vision, vol. 115, no. 3, pp. 211–252, 2015. doi
-
W. Yu, M. Luo, P. Zhou, C. Si, Y. Zhou, X. Wang, J. Feng and S. Yan, “MetaFormer Is Actually What You Need for Vision,” in Proc. IEEE/CVF CVPR, 2022. CVF · arXiv
-
A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto and H. Adam, “MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications,” arXiv:1704.04861, 2017. arXiv
-
M. Sandler, A. Howard, M. Zhu, A. Zhmoginov and L.-C. Chen, “MobileNetV2: Inverted Residuals and Linear Bottlenecks,” in Proc. IEEE/CVF CVPR, 2018. CVF · arXiv
-
K. He, X. Zhang, S. Ren and J. Sun, “Identity Mappings in Deep Residual Networks,” in Proc. European Conference on Computer Vision (ECCV), LNCS, Springer, 2016. doi · arXiv
-
B. Lim, S. Son, H. Kim, S. Nah and K. M. Lee, “Enhanced Deep Residual Networks for Single Image Super-Resolution,” in Proc. IEEE CVPR Workshops (CVPRW), 2017. CVF · arXiv
-
H. Touvron, M. Cord, A. Sablayrolles, G. Synnaeve and H. Jégou, “Going Deeper With Image Transformers,” in Proc. IEEE/CVF ICCV, 2021. CVF
-
L. Chen, X. Chu, X. Zhang and J. Sun, NAFNet official implementation,
basicsr/models/archs/NAFNet_arch.py, GitHub (megvii-research/NAFNet), accessed 2026. code -
Y. Zhang, K. Li, K. Li, L. Wang, B. Zhong and Y. Fu, “Image Super-Resolution Using Very Deep Residual Channel Attention Networks,” in Proc. ECCV, 2018. CVF · arXiv
-
I. Radosavovic, R. P. Kosaraju, R. Girshick, K. He and P. Dollár, “Designing Network Design Spaces,” in Proc. IEEE/CVF CVPR, 2020. CVF · arXiv
-
J. Hu, L. Shen and G. Sun, “Squeeze-and-Excitation Networks,” in Proc. IEEE/CVF CVPR, 2018. CVF · arXiv
-
S. Woo, J. Park, J.-Y. Lee and I. S. Kweon, “CBAM: Convolutional Block Attention Module,” in Proc. ECCV, 2018. CVF · arXiv
-
L. Chen, X. Chu, X. Zhang and J. Sun, “Simple Baselines for Image Restoration,” in Proc. ECCV, LNCS, Springer, 2022. doi · arXiv
-
M.-H. Guo, T.-X. Xu, J.-J. Liu, Z.-N. Liu, P.-T. Jiang, T.-J. Mu, S.-H. Zhang, R. R. Martin, M.-M. Cheng and S.-M. Hu, “Attention Mechanisms in Computer Vision: A Survey,” Computational Visual Media, vol. 8, no. 3, pp. 331–368, 2022. doi · arXiv
-
S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan and M.-H. Yang, “Restormer: Efficient Transformer for High-Resolution Image Restoration,” in Proc. IEEE/CVF CVPR, 2022. CVF · arXiv
-
X. Chen, Z. Li, Y. Pu, Y. Liu, J. Zhou, Y. Qiao and C. Dong, “A Comparative Study of Image Restoration Networks for General Backbone Network Design,” in Proc. ECCV 2024, LNCS, Springer, 2025. doi · arXiv
-
S. Ioffe and C. Szegedy, “Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift,” in Proc. International Conference on Machine Learning (ICML), PMLR 37, 2015. PMLR
-
J. L. Ba, J. R. Kiros and G. E. Hinton, “Layer Normalization,” arXiv:1607.06450, 2016. arXiv
-
D. Ulyanov, A. Vedaldi and V. Lempitsky, “Instance Normalization: The Missing Ingredient for Fast Stylization,” arXiv:1607.08022, 2016. arXiv
-
Y. Wu and K. He, “Group Normalization,” in Proc. ECCV, 2018. CVF · arXiv
-
C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang and W. Shi, “Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network,” in Proc. IEEE CVPR, 2017. CVF
-
X. Wang, K. Yu, S. Wu, J. Gu, Y. Liu, C. Dong, Y. Qiao and C. C. Loy, “ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks,” in Proc. ECCV Workshops, 2018. CVF · arXiv
-
K. Zhang, W. Zuo, Y. Chen, D. Meng and L. Zhang, “Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising,” IEEE Transactions on Image Processing, vol. 26, no. 7, pp. 3142–3155, 2017. doi · arXiv
-
Q. Huynh-Thu and M. Ghanbari, “Scope of Validity of PSNR in Image/Video Quality Assessment,” Electronics Letters, vol. 44, no. 13, pp. 800–801, 2008. doi
-
A. Abdelhamed, S. Lin and M. S. Brown, “A High-Quality Denoising Dataset for Smartphone Cameras,” in Proc. IEEE/CVF CVPR, 2018. CVF
-
S. Nah, T. H. Kim and K. M. Lee, “Deep Multi-Scale Convolutional Neural Network for Dynamic Scene Deblurring,” in Proc. IEEE CVPR, 2017. CVF
-
N. Shazeer, “GLU Variants Improve Transformer,” arXiv:2002.05202, 2020. arXiv
-
Y. N. Dauphin, A. Fan, M. Auli and D. Grangier, “Language Modeling with Gated Convolutional Networks,” in Proc. ICML, PMLR 70, 2017. PMLR
-
D. Hendrycks and K. Gimpel, “Gaussian Error Linear Units (GELUs),” arXiv:1606.08415, 2016. arXiv
-
W. Shi, J. Caballero, F. Huszár, J. Totz, A. P. Aitken, R. Bishop, D. Rueckert and Z. Wang, “Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network,” in Proc. IEEE CVPR, 2016. CVF · arXiv
-
A. Odena, V. Dumoulin and C. Olah, “Deconvolution and Checkerboard Artifacts,” Distill, vol. 1, no. 10, 2016. Distill · doi
-
A. Aitken, C. Ledig, L. Theis, J. Caballero, Z. Wang and W. Shi, “Checkerboard Artifact Free Sub-Pixel Convolution: A Note on Sub-Pixel Convolution, Resize Convolution and Convolution Resize,” arXiv:1707.02937, 2017. arXiv
-
O. Ronneberger, P. Fischer and T. Brox, “U-Net: Convolutional Networks for Biomedical Image Segmentation,” in Proc. Medical Image Computing and Computer-Assisted Intervention (MICCAI), LNCS, Springer, pp. 234–241, 2015. doi · arXiv
-
G. Huang, Z. Liu, L. van der Maaten and K. Q. Weinberger, “Densely Connected Convolutional Networks,” in Proc. IEEE CVPR, 2017. CVF
-
L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy and A. L. Yuille, “DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 40, no. 4, pp. 834–848, 2018. doi · arXiv
-
W. Luo, Y. Li, R. Urtasun and R. Zemel, “Understanding the Effective Receptive Field in Deep Convolutional Neural Networks,” in Advances in Neural Information Processing Systems (NeurIPS), 2016. NeurIPS · arXiv
-
J. Kim, J. K. Lee and K. M. Lee, “Accurate Image Super-Resolution Using Very Deep Convolutional Networks,” in Proc. IEEE CVPR, 2016. CVF · arXiv
-
H. Guo, J. Li, T. Dai, Z. Ouyang, X. Ren and S.-T. Xia, “MambaIR: A Simple Baseline for Image Restoration with State-Space Model,” in Proc. ECCV 2024, LNCS, Springer, 2025. doi · arXiv
-
ITU-T, “Recommendation T.81: Information technology – Digital compression and coding of continuous-tone still images – Requirements and guidelines” (the JPEG standard), 1992. ITU
-
T. Elsken, J. H. Metzen and F. Hutter, “Neural Architecture Search: A Survey,” Journal of Machine Learning Research, vol. 20, 2019. JMLR · arXiv
-
A. Khan, A. Sohail, U. Zahoora and A. S. Qureshi, “A Survey of the Recent Architectures of Deep Convolutional Neural Networks,” Artificial Intelligence Review, vol. 53, no. 8, pp. 5455–5516, 2020. doi · arXiv
-
S. Khan, M. Naseer, M. Hayat, S. W. Zamir, F. S. Khan and M. Shah, “Transformers in Vision: A Survey,” ACM Computing Surveys, vol. 54, no. 10s, 2022. doi · arXiv
-
S. Woo, S. Debnath, R. Hu, X. Chen, Z. Liu, I. S. Kweon and S. Xie, “ConvNeXt V2: Co-Designing and Scaling ConvNets With Masked Autoencoders,” in Proc. IEEE/CVF CVPR, 2023. CVF · arXiv
-
M. Goldblum, H. Souri, R. Ni, M. Shu, V. Prabhu, G. Somepalli, P. Chattopadhyay, M. Ibrahim et al., “Battle of the Backbones: A Large-Scale Comparison of Pretrained Models across Computer Vision Tasks,” in Advances in Neural Information Processing Systems (NeurIPS), 2023. arXiv
-
X. Chen, X. Wang, J. Zhou, Y. Qiao and C. Dong, “Activating More Pixels in Image Super-Resolution Transformer,” in Proc. IEEE/CVF CVPR, 2023. CVF · arXiv
-
H. Guo, Y. Guo, Y. Zha, Y. Zhang, W. Li, T. Dai, S.-T. Xia and Y. Li, “MambaIRv2: Attentive State Space Restoration,” in Proc. IEEE/CVF CVPR, 2025. CVF · arXiv
-
A. Araujo, W. Norris and J. Sim, “Computing Receptive Fields of Convolutional Neural Networks,” Distill, vol. 4, no. 11, 2019. Distill · doi
-
X. Ding, X. Zhang, N. Ma, J. Han, G. Ding and J. Sun, “RepVGG: Making VGG-style ConvNets Great Again,” in Proc. IEEE/CVF CVPR, pp. 13733–13742, 2021. CVF paper
-
S. Zagoruyko and N. Komodakis, “DiracNets: Training Very Deep Neural Networks Without Skip-Connections,” arXiv:1706.00388, 2017. arXiv
-
X. Ding, Y. Guo, G. Ding and J. Han, “ACNet: Strengthening the Kernel Skeletons for Powerful CNN via Asymmetric Convolution Blocks,” in Proc. IEEE/CVF ICCV, 2019. CVF
-
H. Zhao, J. Shi, X. Qi, X. Wang and J. Jia, “Pyramid Scene Parsing Network,” in Proc. IEEE CVPR, 2017. CVF
-
M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth and B. Schiele, “The Cityscapes Dataset for Semantic Urban Scene Understanding,” in Proc. IEEE CVPR, 2016. CVF
-
E. D. Cubuk, B. Zoph, D. Mane, V. Vasudevan and Q. V. Le, “AutoAugment: Learning Augmentation Strategies From Data,” in Proc. IEEE/CVF CVPR, 2019. CVF
-
H. Zhang, M. Cisse, Y. N. Dauphin and D. Lopez-Paz, “mixup: Beyond Empirical Risk Minimization,” in Proc. ICLR, 2018. arXiv