Full search page →

Paper guides

Optional reading guides to the papers behind ML 06 · Representation Learning and Self-Supervision, one page each: the problem, the idea, what it showed and what to check when you read it.

Navigate: this page and this track: Navigate: this page and this track

These guides are optional. Start with the question and original figure, then unfold the evidence, design analysis and reading checkpoint.

  1. SimCLR — what a positive pair teaches a representation

    Can useful visual representations be learned without human class labels by matching transformed views of the same image?

References

  1. Y. Bengio, A. Courville and P. Vincent, “Representation Learning: A Review and New Perspectives,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, no. 8, 2013. Paper · arXiv
  2. J. Yosinski, J. Clune, Y. Bengio and H. Lipson, “How transferable are features in deep neural networks?” NeurIPS, 2014. Paper
  3. X. Liu et al., “Self-supervised Learning: Generative or Contrastive,” IEEE Transactions on Knowledge and Data Engineering, vol. 35, 2023 (online 2021). Paper · arXiv
  4. J. Gui et al., “A Survey on Self-Supervised Learning: Algorithms, Applications, and Future Trends,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024. Paper · arXiv
  5. R. Balestriero et al., “A Cookbook of Self-Supervised Learning,” arXiv:2304.12210, 2023. Paper
  6. A. van den Oord, Y. Li and O. Vinyals, “Representation Learning with Contrastive Predictive Coding,” arXiv:1807.03748, 2018. Paper
  7. T. Chen, S. Kornblith, M. Norouzi and G. Hinton, “A Simple Framework for Contrastive Learning of Visual Representations,” ICML, 2020. Paper
  8. K. He, H. Fan, Y. Wu, S. Xie and R. Girshick, “Momentum Contrast for Unsupervised Visual Representation Learning,” CVPR, 2020. Paper
  9. T. Wang and P. Isola, “Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere,” ICML, 2020. Paper
  10. J.-B. Grill et al., “Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning,” NeurIPS, 2020. Paper
  11. X. Chen and K. He, “Exploring Simple Siamese Representation Learning,” CVPR, 2021. Paper
  12. M. Caron et al., “Emerging Properties in Self-Supervised Vision Transformers,” ICCV, 2021. Paper
  13. J. Zbontar, L. Jing, I. Misra, Y. LeCun and S. Deny, “Barlow Twins: Self-Supervised Learning via Redundancy Reduction,” ICML, 2021. Paper
  14. A. Bardes, J. Ponce and Y. LeCun, “VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning,” ICLR, 2022. Paper
  15. H. Bao, L. Dong, S. Piao and F. Wei, “BEiT: BERT Pre-Training of Image Transformers,” ICLR, 2022. Paper
  16. K. He et al., “Masked Autoencoders Are Scalable Vision Learners,” CVPR, 2022. Paper
  17. M. Assran et al., “Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture,” CVPR, 2023. Paper
  18. M. Assran et al., “V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning,” technical report, 2025. Report
  19. A. Radford et al., “Learning Transferable Visual Models From Natural Language Supervision,” ICML, 2021. Paper
  20. X. Zhai, B. Mustafa, A. Kolesnikov and L. Beyer, “Sigmoid Loss for Language Image Pre-Training,” ICCV, 2023. Paper
  21. M. Oquab et al., “DINOv2: Learning Robust Visual Features without Supervision,” Transactions on Machine Learning Research, 2024. Paper
  22. O. Siméoni et al., “DINOv3,” technical report, 2025. Report
  23. A. Kumar, A. Raghunathan, R. Jones, T. Ma and P. Liang, “Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution,” ICLR, 2022. Paper
  24. S. J. D. Prince, Understanding Deep Learning, MIT Press, 2023. Book and notebooks