Full search page →

ViT — image patches and the role of pretraining scale

Optional · paper guide 1 of 1 for ML 05 · Attention and Vision Transformers. Citation numbers refer to the reference list at the bottom of this page.

Navigate: this page and this track: Navigate: this page and this track

Question and thesis

Can a largely standard Transformer learn useful image representations from a sequence of patches? Dosovitskiy et al. show that it can, with pretraining scale playing a central role [2]. Read patch tokenization, self-attention and residual blocks first. The thesis is not that convolution is always inferior: the paper explicitly investigates how data availability changes the comparison.

Method and figure

Flattened patches become token embeddings with positional information; the Transformer processes their sequence, and a class token feeds the classification head.
Source: Dosovitskiy et al., ICLR 2021 manuscript, arXiv:2010.11929v2. Figure 1, PDF p. 3. Flattened patches become token embeddings with positional information; the Transformer processes their sequence, and a class token feeds the classification head. [2]

Figure 1 starts with a grid of non-overlapping image patches. Flatten each patch, apply a shared learned linear projection, prepend a learnable class token, and add positional embeddings. For an image whose dimensions are divisible by the patch width,

N=HW/P2.N=HW/P^2.

Here HH and WW are image height and width in pixels, PP is the square patch width, and NN is the number of image tokens before adding the class token. The projection maps patch contents to the Transformer hidden width. Position embeddings supply spatial information that a sequence of content vectors alone does not identify.

The encoder alternates attention and multilayer transformations with normalization and residual connections. The final class-token representation goes to the classification head. Smaller patches increase token count, which changes attention cost even when the original image dimensions stay fixed.

Evidence

Figure 3 compares pretraining on ImageNet, ImageNet-21k and JFT-300M. Relative performance changes with the data regime: large ViT models are less competitive when pretrained on less data, while extensive pretraining supports strong transfer. Table 2 reports transfer results with fine-tuning variability. Read the pretraining dataset and compute columns together with downstream accuracy; differing resources prevent treating every row as an isolated architecture comparison.

Design analysis

Convolution builds in locality and weight sharing across positions. ViT retains a patch-based input structure but relies more heavily on data to learn useful spatial relationships. The data-scale study is therefore part of the explanation, not a secondary implementation detail. Also distinguish changing model width from changing patch size: both affect cost, but only the latter directly changes the number of input tokens at fixed resolution.

Reading checkpoint

Halve the patch width while keeping image size fixed. Determine how the number of image tokens changes, then reason about the pairwise attention matrix. Point to where spatial order enters Figure 1, and explain why an attention visualization alone does not establish a causal explanation of the prediction.

Limits and next step

Strong large-scale transfer does not guarantee a favorable small-data or low-compute result. Classification evidence also does not by itself establish dense prediction performance. Continue with ML 06 representation learning and compare supervised pretraining with SimCLR’s training objective: similar downstream use can arise from different learning signals.

Continue reading: SimCLR — what a positive pair teaches a representation

References

  1. A. Vaswani et al., “Attention Is All You Need,” NeurIPS, 2017. arXiv
  2. A. Dosovitskiy et al., “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale,” ICLR, 2021. arXiv
  3. S. Khan, M. Naseer, M. Hayat, S. W. Zamir, F. S. Khan and M. Shah, “Transformers in Vision: A Survey,” ACM Computing Surveys, 2022. arXiv
  4. H. Touvron et al., “Training data-efficient image transformers & distillation through attention,” ICML, 2021. arXiv
  5. Z. Liu et al., “Swin Transformer: Hierarchical Vision Transformer using Shifted Windows,” ICCV, 2021. arXiv
  6. T. Dao, D. Y. Fu, S. Ermon, A. Rudra and C. Ré, “FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness,” NeurIPS, 2022. arXiv
  7. PyTorch, scaled_dot_product_attention and TransformerEncoderLayer documentation. Attention, Encoder layer MultiheadAttention mask semantics
  8. R. Xu et al., “Visual Mamba: A Survey and New Outlooks,” arXiv:2404.18861, 2024. arXiv
  9. W. Yu and X. Wang, “MambaOut: Do We Really Need Mamba for Vision?,” CVPR, 2025. Paper
  10. C. M. Bishop and H. Bishop, Deep Learning: Foundations and Concepts, Springer, 2024, ch. 12. Book
  11. K. Han et al., “A Survey on Vision Transformer,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023. DOI
  12. Y. Liu et al., “A Survey of Visual Transformers,” IEEE Transactions on Neural Networks and Learning Systems, 2024. DOI
  13. L. Papa, P. Russo, I. Amerini and L. Zhou, “A Survey on Efficient Vision Transformers: Algorithms, Techniques, and Performance Benchmarking,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024. DOI
  14. J. L. Ba, J. R. Kiros and G. E. Hinton, “Layer Normalization,” arXiv:1607.06450, 2016. arXiv
  15. R. Xiong et al., “On Layer Normalization in the Transformer Architecture,” ICML, 2020. arXiv
  16. J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo and Y. Liu, “RoFormer: Enhanced Transformer with Rotary Position Embedding,” Neurocomputing, 2024. DOI
  17. P. Shaw, J. Uszkoreit and A. Vaswani, “Self-Attention with Relative Position Representations,” NAACL, 2018. arXiv
  18. M. Raghu, T. Unterthiner, S. Kornblith, C. Zhang and A. Dosovitskiy, “Do Vision Transformers See Like Convolutional Neural Networks?,” NeurIPS, 2021. arXiv
  19. H. Wu et al., “CvT: Introducing Convolutions to Vision Transformers,” ICCV, 2021. arXiv
  20. Z. Dai, H. Liu, Q. V. Le and M. Tan, “CoAtNet: Marrying Convolution and Attention for All Data Sizes,” NeurIPS, 2021. arXiv
  21. D. Bolya et al., “Token Merging: Your ViT But Faster,” ICLR, 2023. arXiv
  22. T. Darcet, M. Oquab, J. Mairal and P. Bojanowski, “Vision Transformers Need Registers,” ICLR, 2024. arXiv
  23. M. Oquab et al., “DINOv2: Learning Robust Visual Features without Supervision,” Transactions on Machine Learning Research, 2024. arXiv
  24. M. Dehghani et al., “Scaling Vision Transformers to 22 Billion Parameters,” ICML, 2023. arXiv
  25. A. Steiner et al., “How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers,” Transactions on Machine Learning Research, 2022. arXiv
  26. D. Hendrycks and K. Gimpel, “Gaussian Error Linear Units (GELUs),” arXiv:1606.08415, 2016. arXiv
  27. H. Touvron, M. Cord and H. Jégou, “DeiT III: Revenge of the ViT,” ECCV, 2022. arXiv
  28. S. Mehta and M. Rastegari, “MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer,” ICLR, 2022. arXiv
  29. T. Dao, “FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning,” ICLR, 2024. arXiv
  30. L. Zhu et al., “Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model,” ICML, 2024. arXiv
  31. Y. Liu et al., “VMamba: Visual State Space Model,” NeurIPS, 2024. arXiv
  32. Z. Liu et al., “Swin Transformer V2: Scaling Up Capacity and Resolution,” CVPR, 2022. arXiv
  33. A. Gu and T. Dao, “Mamba: Linear-Time Sequence Modeling with Selective State Spaces,” COLM, 2024. Author version
  34. T. Dao and A. Gu, “Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality,” ICML, PMLR 235, 2024. Paper