Full search page →

SimCLR — what a positive pair teaches a representation

Optional · paper guide 1 of 1 for ML 06 · Representation Learning and Self-Supervision. Citation numbers refer to the reference list at the bottom of this page.

Navigate: this page and this track: Navigate: this page and this track

Question and thesis

Can useful visual representations be learned without human class labels by matching transformed views of the same image? Chen et al. make the augmentation pipeline, nonlinear projection head and contrastive optimization central design choices [7]. Read vector similarity and cross-entropy first. “Self-supervised” specifies where targets come from; it does not mean that the system has no assumptions about what should remain recognizable.

Method and figure

Two stochastic views share an encoder and projection head. The contrastive objective acts on projected vectors, while the encoder representation is retained for downstream use.
Source: Chen et al., ICML 2020, PMLR 119 proceedings. Figure 2, PDF p. 2. Two stochastic views share an encoder and projection head. The contrastive objective acts on projected vectors, while the encoder representation is retained for downstream use. [7]

Figure 2 draws two independent augmentations of an image. A shared encoder produces representations, and a projection head maps them to the space used for the contrastive loss. With 2N2N views from NN source images, one directed positive-pair loss is

ℓi,j=−log⁡exp⁡(sim⁡(zi,zj)/τ)∑k≠iexp⁡(sim⁡(zi,zk)/τ).\ell_{i,j}=-\log\frac{\exp(\operatorname{sim}(z_i,z_j)/\tau)}{\sum_{k\ne i}\exp(\operatorname{sim}(z_i,z_k)/\tau)}.

Here ziz_i and zjz_j are projected vectors from the same source image, kk indexes the other views in the batch, sim⁡\operatorname{sim} is cosine similarity and τ\tau is a positive temperature. The denominator excludes the anchor itself but includes its positive. Training averages over both directions of each positive pair. The encoder representation before projection is used for downstream evaluation.

Evidence

The paper evaluates frozen representations with a trained linear classifier and also studies limited-label fine-tuning. These protocols answer different questions. Its augmentation experiments show that the composition of cropping and color transformations matters. Figure 8 compares no projection, a linear head and a nonlinear head; the nonlinear head improves the evaluated encoder representation. This is more specific evidence than attributing all gains to contrastive learning as a single ingredient.

Design analysis

The projection space can discard details useful to the contrastive task while allowing the preceding representation to retain information useful elsewhere. Removing the projection head at evaluation therefore follows the paper’s design rather than contradicting training. Batch size, training duration and encoder width also influence the reported results; compare matched settings instead of attaching the strongest headline accuracy to every encoder variant.

Reading checkpoint

Count the candidates in one loss denominator and explain why the positive is present but the anchor is absent. Then propose an augmentation that would change the desired downstream label. What incorrect invariance would treating those views as a positive pair encourage?

Limits and next step

Different source images can share semantic content yet appear as negatives, and aggressive transformations can erase task-relevant details. Linear-probe performance measures accessibility to a restricted classifier, not all possible uses of the representation. Continue to ML 10 and CLIP: identify how replacing a second image view with paired text changes the positive-pair definition and downstream prediction interface.

Continue reading: CLIP — from paired captions to a zero-shot classifier

References

  1. Y. Bengio, A. Courville and P. Vincent, “Representation Learning: A Review and New Perspectives,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, no. 8, 2013. Paper · arXiv
  2. J. Yosinski, J. Clune, Y. Bengio and H. Lipson, “How transferable are features in deep neural networks?” NeurIPS, 2014. Paper
  3. X. Liu et al., “Self-supervised Learning: Generative or Contrastive,” IEEE Transactions on Knowledge and Data Engineering, vol. 35, 2023 (online 2021). Paper · arXiv
  4. J. Gui et al., “A Survey on Self-Supervised Learning: Algorithms, Applications, and Future Trends,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024. Paper · arXiv
  5. R. Balestriero et al., “A Cookbook of Self-Supervised Learning,” arXiv:2304.12210, 2023. Paper
  6. A. van den Oord, Y. Li and O. Vinyals, “Representation Learning with Contrastive Predictive Coding,” arXiv:1807.03748, 2018. Paper
  7. T. Chen, S. Kornblith, M. Norouzi and G. Hinton, “A Simple Framework for Contrastive Learning of Visual Representations,” ICML, 2020. Paper
  8. K. He, H. Fan, Y. Wu, S. Xie and R. Girshick, “Momentum Contrast for Unsupervised Visual Representation Learning,” CVPR, 2020. Paper
  9. T. Wang and P. Isola, “Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere,” ICML, 2020. Paper
  10. J.-B. Grill et al., “Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning,” NeurIPS, 2020. Paper
  11. X. Chen and K. He, “Exploring Simple Siamese Representation Learning,” CVPR, 2021. Paper
  12. M. Caron et al., “Emerging Properties in Self-Supervised Vision Transformers,” ICCV, 2021. Paper
  13. J. Zbontar, L. Jing, I. Misra, Y. LeCun and S. Deny, “Barlow Twins: Self-Supervised Learning via Redundancy Reduction,” ICML, 2021. Paper
  14. A. Bardes, J. Ponce and Y. LeCun, “VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning,” ICLR, 2022. Paper
  15. H. Bao, L. Dong, S. Piao and F. Wei, “BEiT: BERT Pre-Training of Image Transformers,” ICLR, 2022. Paper
  16. K. He et al., “Masked Autoencoders Are Scalable Vision Learners,” CVPR, 2022. Paper
  17. M. Assran et al., “Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture,” CVPR, 2023. Paper
  18. M. Assran et al., “V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning,” technical report, 2025. Report
  19. A. Radford et al., “Learning Transferable Visual Models From Natural Language Supervision,” ICML, 2021. Paper
  20. X. Zhai, B. Mustafa, A. Kolesnikov and L. Beyer, “Sigmoid Loss for Language Image Pre-Training,” ICCV, 2023. Paper
  21. M. Oquab et al., “DINOv2: Learning Robust Visual Features without Supervision,” Transactions on Machine Learning Research, 2024. Paper
  22. O. Siméoni et al., “DINOv3,” technical report, 2025. Report
  23. A. Kumar, A. Raghunathan, R. Jones, T. Ma and P. Liang, “Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution,” ICLR, 2022. Paper
  24. S. J. D. Prince, Understanding Deep Learning, MIT Press, 2023. Book and notebooks