完整搜尋頁面 →

SimCLR:正樣本對如何塑造表示

選讀・ML 06・表示學習與自監督 的論文導讀(第 1/1 篇)。引用編號對應本頁底部的參考文獻。

導覽:本頁與本系列: 導覽:本頁與本系列

問題與論點

不使用人工類別標籤,能否透過配對同一影像的變換視圖,學到有用視覺表示?Chen 等人把增強流程、非線性投影頭與對比最佳化列為核心設計 [7]。先讀向量相似度與交叉熵。「自監督」說明目標從哪裡來,不代表系統沒有「哪些內容應保持可辨識」的假設。

方法與圖解

兩個隨機視圖共享編碼器與投影頭;對比目標作用於投影向量,下游則保留編碼器表示。
來源:Chen et al., ICML 2020, PMLR 119 proceedings. 圖 2,PDF 第 2 頁。兩個隨機視圖共享編碼器與投影頭;對比目標作用於投影向量,下游則保留編碼器表示。 [7]

圖 2 對同一影像獨立抽取兩種增強。共享編碼器產生表示,投影頭再映到計算對比損失的空間。由 NN 張來源影像得到 2N2N 個視圖時,單向正樣本對損失為

ℓi,j=−log⁡exp⁡(sim⁡(zi,zj)/τ)∑k≠iexp⁡(sim⁡(zi,zk)/τ).\ell_{i,j}=-\log\frac{\exp(\operatorname{sim}(z_i,z_j)/\tau)}{\sum_{k\ne i}\exp(\operatorname{sim}(z_i,z_k)/\tau)}.

其中 ziz_i、zjz_j 是同一來源影像的投影向量,kk 遍歷批次中其他視圖,sim⁡\operatorname{sim} 為餘弦相似度,τ\tau 為正溫度。分母排除錨點自己,但包含正樣本。訓練會平均每對正樣本的兩個方向;下游評估使用投影前的編碼器表示。

證據

論文以凍結表示搭配可訓練線性分類器評估,也研究少標籤微調;兩種協定回答不同問題。增強實驗顯示,裁切與色彩變換的組合很重要。圖 8 比較無投影、線性頭與非線性頭,非線性頭改善了被評估的編碼器表示。這比把所有進步都歸給單一「對比學習」標籤更具體。

設計分析

投影空間可以捨去對比任務不需要的細節,同時讓前一層表示保留其他任務可能需要的資訊。因此評估時移除投影頭符合論文設計,並非與訓練矛盾。批次大小、訓練長度與編碼器寬度也影響結果;應比較相同設定,不要把最高的摘要準確度套到每種編碼器。

閱讀檢查點

算出單一損失分母有多少候選,說明為何包含正樣本卻排除錨點。再提出一種會改變下游標籤的增強;若仍把兩個視圖視為正樣本對,會鼓勵哪種不正確的不變性?

限制與下一步

不同來源影像可能共享語意,卻被當成負樣本;強增強也可能刪除任務所需細節。線性探測量測受限分類器能否取用表示,不代表表示的所有可能用途。接著讀 ML 10 與 CLIP:把第二張影像視圖換成配對文字,會如何改變正樣本定義與下游預測介面?

接著閱讀: CLIP:從影像文字配對到零樣本分類器

參考文獻

  1. Y. Bengio, A. Courville and P. Vincent, “Representation Learning: A Review and New Perspectives,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, no. 8, 2013. Paper · arXiv
  2. J. Yosinski, J. Clune, Y. Bengio and H. Lipson, “How transferable are features in deep neural networks?” NeurIPS, 2014. Paper
  3. X. Liu et al., “Self-supervised Learning: Generative or Contrastive,” IEEE Transactions on Knowledge and Data Engineering, vol. 35, 2023 (online 2021). Paper · arXiv
  4. J. Gui et al., “A Survey on Self-Supervised Learning: Algorithms, Applications, and Future Trends,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024. Paper · arXiv
  5. R. Balestriero et al., “A Cookbook of Self-Supervised Learning,” arXiv:2304.12210, 2023. Paper
  6. A. van den Oord, Y. Li and O. Vinyals, “Representation Learning with Contrastive Predictive Coding,” arXiv:1807.03748, 2018. Paper
  7. T. Chen, S. Kornblith, M. Norouzi and G. Hinton, “A Simple Framework for Contrastive Learning of Visual Representations,” ICML, 2020. Paper
  8. K. He, H. Fan, Y. Wu, S. Xie and R. Girshick, “Momentum Contrast for Unsupervised Visual Representation Learning,” CVPR, 2020. Paper
  9. T. Wang and P. Isola, “Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere,” ICML, 2020. Paper
  10. J.-B. Grill et al., “Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning,” NeurIPS, 2020. Paper
  11. X. Chen and K. He, “Exploring Simple Siamese Representation Learning,” CVPR, 2021. Paper
  12. M. Caron et al., “Emerging Properties in Self-Supervised Vision Transformers,” ICCV, 2021. Paper
  13. J. Zbontar, L. Jing, I. Misra, Y. LeCun and S. Deny, “Barlow Twins: Self-Supervised Learning via Redundancy Reduction,” ICML, 2021. Paper
  14. A. Bardes, J. Ponce and Y. LeCun, “VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning,” ICLR, 2022. Paper
  15. H. Bao, L. Dong, S. Piao and F. Wei, “BEiT: BERT Pre-Training of Image Transformers,” ICLR, 2022. Paper
  16. K. He et al., “Masked Autoencoders Are Scalable Vision Learners,” CVPR, 2022. Paper
  17. M. Assran et al., “Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture,” CVPR, 2023. Paper
  18. M. Assran et al., “V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning,” technical report, 2025. Report
  19. A. Radford et al., “Learning Transferable Visual Models From Natural Language Supervision,” ICML, 2021. Paper
  20. X. Zhai, B. Mustafa, A. Kolesnikov and L. Beyer, “Sigmoid Loss for Language Image Pre-Training,” ICCV, 2023. Paper
  21. M. Oquab et al., “DINOv2: Learning Robust Visual Features without Supervision,” Transactions on Machine Learning Research, 2024. Paper
  22. O. Siméoni et al., “DINOv3,” technical report, 2025. Report
  23. A. Kumar, A. Raghunathan, R. Jones, T. Ma and P. Liang, “Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution,” ICLR, 2022. Paper
  24. S. J. D. Prince, Understanding Deep Learning, MIT Press, 2023. Book and notebooks