ViT — image patches and the role of pretraining scale
Optional · paper guide 1 of 1 for ML 05 · Attention and Vision Transformers. Citation numbers refer to the reference list at the bottom of this page.
Navigate: this page and this track: Navigate: this page and this track
Question and thesis
Can a largely standard Transformer learn useful image representations from a sequence of patches? Dosovitskiy et al. show that it can, with pretraining scale playing a central role [2]. Read patch tokenization, self-attention and residual blocks first. The thesis is not that convolution is always inferior: the paper explicitly investigates how data availability changes the comparison.
Method and figure

Figure 1 starts with a grid of non-overlapping image patches. Flatten each patch, apply a shared learned linear projection, prepend a learnable class token, and add positional embeddings. For an image whose dimensions are divisible by the patch width,
Here and are image height and width in pixels, is the square patch width, and is the number of image tokens before adding the class token. The projection maps patch contents to the Transformer hidden width. Position embeddings supply spatial information that a sequence of content vectors alone does not identify.
The encoder alternates attention and multilayer transformations with normalization and residual connections. The final class-token representation goes to the classification head. Smaller patches increase token count, which changes attention cost even when the original image dimensions stay fixed.
Evidence
Figure 3 compares pretraining on ImageNet, ImageNet-21k and JFT-300M. Relative performance changes with the data regime: large ViT models are less competitive when pretrained on less data, while extensive pretraining supports strong transfer. Table 2 reports transfer results with fine-tuning variability. Read the pretraining dataset and compute columns together with downstream accuracy; differing resources prevent treating every row as an isolated architecture comparison.
Design analysis
Convolution builds in locality and weight sharing across positions. ViT retains a patch-based input structure but relies more heavily on data to learn useful spatial relationships. The data-scale study is therefore part of the explanation, not a secondary implementation detail. Also distinguish changing model width from changing patch size: both affect cost, but only the latter directly changes the number of input tokens at fixed resolution.
Reading checkpoint
Halve the patch width while keeping image size fixed. Determine how the number of image tokens changes, then reason about the pairwise attention matrix. Point to where spatial order enters Figure 1, and explain why an attention visualization alone does not establish a causal explanation of the prediction.
Limits and next step
Strong large-scale transfer does not guarantee a favorable small-data or low-compute result. Classification evidence also does not by itself establish dense prediction performance. Continue with ML 06 representation learning and compare supervised pretraining with SimCLR’s training objective: similar downstream use can arise from different learning signals.
Continue reading: SimCLR — what a positive pair teaches a representation
References
- A. Vaswani et al., “Attention Is All You Need,” NeurIPS, 2017. arXiv
- A. Dosovitskiy et al., “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale,” ICLR, 2021. arXiv
- S. Khan, M. Naseer, M. Hayat, S. W. Zamir, F. S. Khan and M. Shah, “Transformers in Vision: A Survey,” ACM Computing Surveys, 2022. arXiv
- H. Touvron et al., “Training data-efficient image transformers & distillation through attention,” ICML, 2021. arXiv
- Z. Liu et al., “Swin Transformer: Hierarchical Vision Transformer using Shifted Windows,” ICCV, 2021. arXiv
- T. Dao, D. Y. Fu, S. Ermon, A. Rudra and C. Ré, “FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness,” NeurIPS, 2022. arXiv
- PyTorch,
scaled_dot_product_attentionandTransformerEncoderLayerdocumentation. Attention, Encoder layer MultiheadAttention mask semantics - R. Xu et al., “Visual Mamba: A Survey and New Outlooks,” arXiv:2404.18861, 2024. arXiv
- W. Yu and X. Wang, “MambaOut: Do We Really Need Mamba for Vision?,” CVPR, 2025. Paper
- C. M. Bishop and H. Bishop, Deep Learning: Foundations and Concepts, Springer, 2024, ch. 12. Book
- K. Han et al., “A Survey on Vision Transformer,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023. DOI
- Y. Liu et al., “A Survey of Visual Transformers,” IEEE Transactions on Neural Networks and Learning Systems, 2024. DOI
- L. Papa, P. Russo, I. Amerini and L. Zhou, “A Survey on Efficient Vision Transformers: Algorithms, Techniques, and Performance Benchmarking,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024. DOI
- J. L. Ba, J. R. Kiros and G. E. Hinton, “Layer Normalization,” arXiv:1607.06450, 2016. arXiv
- R. Xiong et al., “On Layer Normalization in the Transformer Architecture,” ICML, 2020. arXiv
- J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo and Y. Liu, “RoFormer: Enhanced Transformer with Rotary Position Embedding,” Neurocomputing, 2024. DOI
- P. Shaw, J. Uszkoreit and A. Vaswani, “Self-Attention with Relative Position Representations,” NAACL, 2018. arXiv
- M. Raghu, T. Unterthiner, S. Kornblith, C. Zhang and A. Dosovitskiy, “Do Vision Transformers See Like Convolutional Neural Networks?,” NeurIPS, 2021. arXiv
- H. Wu et al., “CvT: Introducing Convolutions to Vision Transformers,” ICCV, 2021. arXiv
- Z. Dai, H. Liu, Q. V. Le and M. Tan, “CoAtNet: Marrying Convolution and Attention for All Data Sizes,” NeurIPS, 2021. arXiv
- D. Bolya et al., “Token Merging: Your ViT But Faster,” ICLR, 2023. arXiv
- T. Darcet, M. Oquab, J. Mairal and P. Bojanowski, “Vision Transformers Need Registers,” ICLR, 2024. arXiv
- M. Oquab et al., “DINOv2: Learning Robust Visual Features without Supervision,” Transactions on Machine Learning Research, 2024. arXiv
- M. Dehghani et al., “Scaling Vision Transformers to 22 Billion Parameters,” ICML, 2023. arXiv
- A. Steiner et al., “How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers,” Transactions on Machine Learning Research, 2022. arXiv
- D. Hendrycks and K. Gimpel, “Gaussian Error Linear Units (GELUs),” arXiv:1606.08415, 2016. arXiv
- H. Touvron, M. Cord and H. Jégou, “DeiT III: Revenge of the ViT,” ECCV, 2022. arXiv
- S. Mehta and M. Rastegari, “MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer,” ICLR, 2022. arXiv
- T. Dao, “FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning,” ICLR, 2024. arXiv
- L. Zhu et al., “Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model,” ICML, 2024. arXiv
- Y. Liu et al., “VMamba: Visual State Space Model,” NeurIPS, 2024. arXiv
- Z. Liu et al., “Swin Transformer V2: Scaling Up Capacity and Resolution,” CVPR, 2022. arXiv
- A. Gu and T. Dao, “Mamba: Linear-Time Sequence Modeling with Selective State Spaces,” COLM, 2024. Author version
- T. Dao and A. Gu, “Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality,” ICML, PMLR 235, 2024. Paper