完整搜尋頁面 →

CLIP:從影像文字配對到零樣本分類器

選讀・ML 10・基礎模型與視覺語言學習 的論文導讀(第 1/1 篇)。引用編號對應本頁底部的參考文獻。

導覽:本頁與本系列: 導覽:本頁與本系列

問題與論點

自然語言監督能否讓影像編碼器適用於新分類任務,而不必重新擬合任務專用分類器?Radford 等人以網路影像文字配對訓練兩個編碼器,再由文字描述建立類別表示 [3]。先讀對比學習與餘弦相似度。零樣本指下游遷移,不代表預訓練完全沒有監督。

方法與圖解

三階段分別訓練配對嵌入、建立文字類別向量,再將影像與類別向量比較;下游類別清單會改變預測任務。
來源:Radford et al., ICML 2021, PMLR 139 proceedings. 圖 1,PDF 第 2 頁。三階段分別訓練配對嵌入、建立文字類別向量,再將影像與類別向量比較;下游類別清單會改變預測任務。 [3]

圖 1 區分對比預訓練、分類器建構及零樣本預測。對一批配對資料,相似度為

sij=exp⁡(a) ui⊤vj.s_{ij}=\exp(a)\,u_i^\top v_j.

其中 uiu_i 為第 ii 筆正規化影像嵌入,vjv_j 為第 jj 個圖說的正規化文字嵌入,aa 為可學習對數尺度。對稱交叉熵在相似度矩陣每列及每欄辨識正確配對;非對角配對則作為批次內其他候選。

分類時,將加入提示的類別名稱送入文字編碼器,再與影像向量比較。類別清單定義候選集合,文字模板提供語境;多模板表示平均則是另一項評估選擇。兩者都不需要使用該下游任務的標記範例,訓練新的影像分類器。

證據

圖 4 在 27 個資料集上,比較零樣本 CLIP 與凍結 ResNet-50 特徵上的監督式邏輯斯迴歸。CLIP 在其中 16 項比較勝出,但不同資料集的表現差異很大。這不是擊敗所有完整微調專用模型的主張。論文也研究提示設計與集成,說明精確文字介面應列入評估協定,不能視為無關的排版。

設計分析

配對文字提供的正樣本定義,不同於 SimCLR 的兩張增強影像。它能描述概念而不受單一固定分類表限制,但網路圖說對影像內容的描述不完整且含雜訊。推論時改候選類別或措辭,也會改變比較。因此相似度排序取決於文字介面,不是視覺概念為真的固有機率。

閱讀檢查點

指出批次相似度矩陣中的正樣本位置。再比較零樣本文字原型,以及用標籤訓練的凍結影像特徵分類器:各自有哪些參數或決策規則使用了下游標籤?為何分數應分開標示?

限制與下一步

預訓練涵蓋範圍、資料重疊、語言選擇與社會偏誤都會影響遷移。即使不做梯度更新,提示挑選仍可能納入人工任務知識。相似度不自動等於校準信心,也不是可靠的分布外警報。接著讀 ML 11 與校準導讀,再判斷 softmax 數值能否視為預測正確的機率。

接著閱讀: Temperature Scaling:不改類別的信心校準

參考文獻

  1. R. Bommasani et al., “On the Opportunities and Risks of Foundation Models,” Stanford CRFM report, arXiv:2108.07258, 2021; revised 2022. arXiv
  2. D. Caffagni et al., “The Revolution of Multimodal Large Language Models: A Survey,” Findings of ACL, 2024. arXiv
  3. A. Radford et al., “Learning Transferable Visual Models From Natural Language Supervision,” ICML, 2021. arXiv
  4. X. Zhai, B. Mustafa, A. Kolesnikov and L. Beyer, “Sigmoid Loss for Language Image Pre-Training,” ICCV, 2023. arXiv
  5. A. Kirillov et al., “Segment Anything,” ICCV, 2023. arXiv
  6. N. Ravi et al., “SAM 2: Segment Anything in Images and Videos,” ICLR, 2025. arXiv
  7. N. Carion et al., “SAM 3: Segment Anything with Concepts,” arXiv:2511.16719, 2025; revised March 2026. arXiv
  8. H. Liu, C. Li, Q. Wu and Y. J. Lee, “Visual Instruction Tuning,” NeurIPS, 2023. arXiv
  9. S. Bai et al., “Qwen3-VL Technical Report,” arXiv:2511.21631, 2025. arXiv
  10. E. J. Hu et al., “LoRA: Low-Rank Adaptation of Large Language Models,” ICLR, 2022. arXiv
  11. M. Assran et al., “V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning,” arXiv:2506.09985, 2025. arXiv
  12. J. Wang et al., “VGGT: Visual Geometry Grounded Transformer,” CVPR, 2025. arXiv
  13. L. Han et al., “A Survey of Generative Categories and Techniques in Multimodal Generative Models,” arXiv:2506.10016, 2025. arXiv
  14. C. M. Bishop and H. Bishop, Deep Learning: Foundations and Concepts, Springer, 2024. Book
  15. M. Oquab et al., “DINOv2: Learning Robust Visual Features without Supervision,” Transactions on Machine Learning Research, 2024. arXiv
  16. O. Siméoni et al., “DINOv3,” arXiv:2508.10104, 2025. arXiv
  17. J. Kaplan et al., “Scaling Laws for Neural Language Models,” arXiv:2001.08361, 2020. arXiv
  18. J. Hoffmann et al., “Training Compute-Optimal Large Language Models,” arXiv:2203.15556, 2022; published as “An empirical analysis of compute-optimal large language model training,” NeurIPS, 2022. arXiv
  19. X. Zhai, A. Kolesnikov, N. Houlsby and L. Beyer, “Scaling Vision Transformers,” CVPR, 2022. arXiv
  20. M. Cherti et al., “Reproducible Scaling Laws for Contrastive Language-Image Learning,” CVPR, 2023. arXiv
  21. A. Dosovitskiy et al., “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale,” ICLR, 2021. arXiv
  22. K. He, X. Chen, S. Xie, Y. Li, P. Dollár and R. Girshick, “Masked Autoencoders Are Scalable Vision Learners,” CVPR, 2022. arXiv
  23. H. Bao, L. Dong, S. Piao and F. Wei, “BEiT: BERT Pre-Training of Image Transformers,” ICLR, 2022. arXiv
  24. M. Caron et al., “Emerging Properties in Self-Supervised Vision Transformers,” ICCV, 2021. arXiv
  25. C. Jia et al., “Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision,” ICML, 2021. arXiv
  26. J. Li, D. Li, S. Savarese and S. Hoi, “BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models,” ICML, 2023. PMLR
  27. J.-B. Alayrac et al., “Flamingo: a Visual Language Model for Few-Shot Learning,” NeurIPS, 2022. arXiv
  28. H. Liu, C. Li, Y. Li and Y. J. Lee, “Improved Baselines with Visual Instruction Tuning,” CVPR, 2024. arXiv
  29. N. Houlsby et al., “Parameter-Efficient Transfer Learning for NLP,” ICML, 2019. PMLR
  30. M. Jia et al., “Visual Prompt Tuning,” ECCV, 2022. arXiv
  31. T. Dettmers, A. Pagnoni, A. Holtzman and L. Zettlemoyer, “QLoRA: Efficient Finetuning of Quantized LLMs,” NeurIPS, 2023. arXiv
  32. A. Kumar, A. Raghunathan, R. Jones, T. Ma and P. Liang, “Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution,” ICLR, 2022. arXiv
  33. M. Wortsman et al., “Robust Fine-Tuning of Zero-Shot Models,” CVPR, 2022. arXiv
  34. C. Schuhmann et al., “LAION-5B: An Open Large-Scale Dataset for Training Next Generation Image-Text Models,” NeurIPS Datasets and Benchmarks Track, 2022. arXiv
  35. A. Birhane, V. U. Prabhu and E. Kahembwe, “Multimodal Datasets: Misogyny, Pornography, and Malignant Stereotypes,” arXiv:2110.01963, 2021. arXiv
  36. P. Mayilvahanan, T. Wiedemer, E. Rusak, M. Bethge and W. Brendel, “Does CLIP’s Generalization Performance Mainly Stem from High Train-Test Similarity?,” ICLR, 2024. arXiv
  37. Y. Li et al., “Evaluating Object Hallucination in Large Vision-Language Models,” EMNLP, 2023. arXiv
  38. X. Yue et al., “MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI,” CVPR, 2024. arXiv
  39. M. Awais et al., “Foundational Models Defining a New Era in Vision: A Survey and Outlook,” arXiv:2307.13721, 2023. arXiv
  40. S. Yin et al., “A Survey on Multimodal Large Language Models,” National Science Review, 2024. arXiv
  41. J. Zhang, J. Huang, S. Jin and S. Lu, “Vision-Language Models for Vision Tasks: A Survey,” IEEE TPAMI, vol. 46, no. 8, 2024. arXiv
  42. L. Xu et al., “Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment,” arXiv:2312.12148, 2023. arXiv
  43. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li and L. Fei-Fei, “ImageNet: A Large-Scale Hierarchical Image Database,” CVPR, 2009, pp. 248–255. doi
  44. J. Devlin, M.-W. Chang, K. Lee and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” NAACL-HLT, 2019, pp. 4171–4186. doi
  45. OpenAI, “GPT-4 Technical Report,” arXiv:2303.08774, 2023. arXiv