SimCLR — what a positive pair teaches a representation
Optional · paper guide 1 of 1 for ML 06 · Representation Learning and Self-Supervision. Citation numbers refer to the reference list at the bottom of this page.
Navigate: this page and this track: Navigate: this page and this track
Question and thesis
Can useful visual representations be learned without human class labels by matching transformed views of the same image? Chen et al. make the augmentation pipeline, nonlinear projection head and contrastive optimization central design choices [7]. Read vector similarity and cross-entropy first. “Self-supervised” specifies where targets come from; it does not mean that the system has no assumptions about what should remain recognizable.
Method and figure

Figure 2 draws two independent augmentations of an image. A shared encoder produces representations, and a projection head maps them to the space used for the contrastive loss. With views from source images, one directed positive-pair loss is
Here and are projected vectors from the same source image, indexes the other views in the batch, is cosine similarity and is a positive temperature. The denominator excludes the anchor itself but includes its positive. Training averages over both directions of each positive pair. The encoder representation before projection is used for downstream evaluation.
Evidence
The paper evaluates frozen representations with a trained linear classifier and also studies limited-label fine-tuning. These protocols answer different questions. Its augmentation experiments show that the composition of cropping and color transformations matters. Figure 8 compares no projection, a linear head and a nonlinear head; the nonlinear head improves the evaluated encoder representation. This is more specific evidence than attributing all gains to contrastive learning as a single ingredient.
Design analysis
The projection space can discard details useful to the contrastive task while allowing the preceding representation to retain information useful elsewhere. Removing the projection head at evaluation therefore follows the paper’s design rather than contradicting training. Batch size, training duration and encoder width also influence the reported results; compare matched settings instead of attaching the strongest headline accuracy to every encoder variant.
Reading checkpoint
Count the candidates in one loss denominator and explain why the positive is present but the anchor is absent. Then propose an augmentation that would change the desired downstream label. What incorrect invariance would treating those views as a positive pair encourage?
Limits and next step
Different source images can share semantic content yet appear as negatives, and aggressive transformations can erase task-relevant details. Linear-probe performance measures accessibility to a restricted classifier, not all possible uses of the representation. Continue to ML 10 and CLIP: identify how replacing a second image view with paired text changes the positive-pair definition and downstream prediction interface.
Continue reading: CLIP — from paired captions to a zero-shot classifier
References
- Y. Bengio, A. Courville and P. Vincent, “Representation Learning: A Review and New Perspectives,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, no. 8, 2013. Paper · arXiv
- J. Yosinski, J. Clune, Y. Bengio and H. Lipson, “How transferable are features in deep neural networks?” NeurIPS, 2014. Paper
- X. Liu et al., “Self-supervised Learning: Generative or Contrastive,” IEEE Transactions on Knowledge and Data Engineering, vol. 35, 2023 (online 2021). Paper · arXiv
- J. Gui et al., “A Survey on Self-Supervised Learning: Algorithms, Applications, and Future Trends,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024. Paper · arXiv
- R. Balestriero et al., “A Cookbook of Self-Supervised Learning,” arXiv:2304.12210, 2023. Paper
- A. van den Oord, Y. Li and O. Vinyals, “Representation Learning with Contrastive Predictive Coding,” arXiv:1807.03748, 2018. Paper
- T. Chen, S. Kornblith, M. Norouzi and G. Hinton, “A Simple Framework for Contrastive Learning of Visual Representations,” ICML, 2020. Paper
- K. He, H. Fan, Y. Wu, S. Xie and R. Girshick, “Momentum Contrast for Unsupervised Visual Representation Learning,” CVPR, 2020. Paper
- T. Wang and P. Isola, “Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere,” ICML, 2020. Paper
- J.-B. Grill et al., “Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning,” NeurIPS, 2020. Paper
- X. Chen and K. He, “Exploring Simple Siamese Representation Learning,” CVPR, 2021. Paper
- M. Caron et al., “Emerging Properties in Self-Supervised Vision Transformers,” ICCV, 2021. Paper
- J. Zbontar, L. Jing, I. Misra, Y. LeCun and S. Deny, “Barlow Twins: Self-Supervised Learning via Redundancy Reduction,” ICML, 2021. Paper
- A. Bardes, J. Ponce and Y. LeCun, “VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning,” ICLR, 2022. Paper
- H. Bao, L. Dong, S. Piao and F. Wei, “BEiT: BERT Pre-Training of Image Transformers,” ICLR, 2022. Paper
- K. He et al., “Masked Autoencoders Are Scalable Vision Learners,” CVPR, 2022. Paper
- M. Assran et al., “Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture,” CVPR, 2023. Paper
- M. Assran et al., “V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning,” technical report, 2025. Report
- A. Radford et al., “Learning Transferable Visual Models From Natural Language Supervision,” ICML, 2021. Paper
- X. Zhai, B. Mustafa, A. Kolesnikov and L. Beyer, “Sigmoid Loss for Language Image Pre-Training,” ICCV, 2023. Paper
- M. Oquab et al., “DINOv2: Learning Robust Visual Features without Supervision,” Transactions on Machine Learning Research, 2024. Paper
- O. Siméoni et al., “DINOv3,” technical report, 2025. Report
- A. Kumar, A. Raghunathan, R. Jones, T. Ma and P. Liang, “Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution,” ICLR, 2022. Paper
- S. J. D. Prince, Understanding Deep Learning, MIT Press, 2023. Book and notebooks