CLIP:從影像文字配對到零樣本分類器
選讀・ML 10・基礎模型與視覺語言學習 的論文導讀(第 1/1 篇)。引用編號對應本頁底部的參考文獻。
導覽:本頁與本系列: 導覽:本頁與本系列
問題與論點
自然語言監督能否讓影像編碼器適用於新分類任務,而不必重新擬合任務專用分類器?Radford 等人以網路影像文字配對訓練兩個編碼器,再由文字描述建立類別表示 [3]。先讀對比學習與餘弦相似度。零樣本指下游遷移,不代表預訓練完全沒有監督。
方法與圖解

圖 1 區分對比預訓練、分類器建構及零樣本預測。對一批配對資料,相似度為
其中 為第 筆正規化影像嵌入, 為第 個圖說的正規化文字嵌入, 為可學習對數尺度。對稱交叉熵在相似度矩陣每列及每欄辨識正確配對;非對角配對則作為批次內其他候選。
分類時,將加入提示的類別名稱送入文字編碼器,再與影像向量比較。類別清單定義候選集合,文字模板提供語境;多模板表示平均則是另一項評估選擇。兩者都不需要使用該下游任務的標記範例,訓練新的影像分類器。
證據
圖 4 在 27 個資料集上,比較零樣本 CLIP 與凍結 ResNet-50 特徵上的監督式邏輯斯迴歸。CLIP 在其中 16 項比較勝出,但不同資料集的表現差異很大。這不是擊敗所有完整微調專用模型的主張。論文也研究提示設計與集成,說明精確文字介面應列入評估協定,不能視為無關的排版。
設計分析
配對文字提供的正樣本定義,不同於 SimCLR 的兩張增強影像。它能描述概念而不受單一固定分類表限制,但網路圖說對影像內容的描述不完整且含雜訊。推論時改候選類別或措辭,也會改變比較。因此相似度排序取決於文字介面,不是視覺概念為真的固有機率。
閱讀檢查點
指出批次相似度矩陣中的正樣本位置。再比較零樣本文字原型,以及用標籤訓練的凍結影像特徵分類器:各自有哪些參數或決策規則使用了下游標籤?為何分數應分開標示?
限制與下一步
預訓練涵蓋範圍、資料重疊、語言選擇與社會偏誤都會影響遷移。即使不做梯度更新,提示挑選仍可能納入人工任務知識。相似度不自動等於校準信心,也不是可靠的分布外警報。接著讀 ML 11 與校準導讀,再判斷 softmax 數值能否視為預測正確的機率。
接著閱讀: Temperature Scaling:不改類別的信心校準
參考文獻
- R. Bommasani et al., “On the Opportunities and Risks of Foundation Models,” Stanford CRFM report, arXiv:2108.07258, 2021; revised 2022. arXiv
- D. Caffagni et al., “The Revolution of Multimodal Large Language Models: A Survey,” Findings of ACL, 2024. arXiv
- A. Radford et al., “Learning Transferable Visual Models From Natural Language Supervision,” ICML, 2021. arXiv
- X. Zhai, B. Mustafa, A. Kolesnikov and L. Beyer, “Sigmoid Loss for Language Image Pre-Training,” ICCV, 2023. arXiv
- A. Kirillov et al., “Segment Anything,” ICCV, 2023. arXiv
- N. Ravi et al., “SAM 2: Segment Anything in Images and Videos,” ICLR, 2025. arXiv
- N. Carion et al., “SAM 3: Segment Anything with Concepts,” arXiv:2511.16719, 2025; revised March 2026. arXiv
- H. Liu, C. Li, Q. Wu and Y. J. Lee, “Visual Instruction Tuning,” NeurIPS, 2023. arXiv
- S. Bai et al., “Qwen3-VL Technical Report,” arXiv:2511.21631, 2025. arXiv
- E. J. Hu et al., “LoRA: Low-Rank Adaptation of Large Language Models,” ICLR, 2022. arXiv
- M. Assran et al., “V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning,” arXiv:2506.09985, 2025. arXiv
- J. Wang et al., “VGGT: Visual Geometry Grounded Transformer,” CVPR, 2025. arXiv
- L. Han et al., “A Survey of Generative Categories and Techniques in Multimodal Generative Models,” arXiv:2506.10016, 2025. arXiv
- C. M. Bishop and H. Bishop, Deep Learning: Foundations and Concepts, Springer, 2024. Book
- M. Oquab et al., “DINOv2: Learning Robust Visual Features without Supervision,” Transactions on Machine Learning Research, 2024. arXiv
- O. Siméoni et al., “DINOv3,” arXiv:2508.10104, 2025. arXiv
- J. Kaplan et al., “Scaling Laws for Neural Language Models,” arXiv:2001.08361, 2020. arXiv
- J. Hoffmann et al., “Training Compute-Optimal Large Language Models,” arXiv:2203.15556, 2022; published as “An empirical analysis of compute-optimal large language model training,” NeurIPS, 2022. arXiv
- X. Zhai, A. Kolesnikov, N. Houlsby and L. Beyer, “Scaling Vision Transformers,” CVPR, 2022. arXiv
- M. Cherti et al., “Reproducible Scaling Laws for Contrastive Language-Image Learning,” CVPR, 2023. arXiv
- A. Dosovitskiy et al., “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale,” ICLR, 2021. arXiv
- K. He, X. Chen, S. Xie, Y. Li, P. Dollár and R. Girshick, “Masked Autoencoders Are Scalable Vision Learners,” CVPR, 2022. arXiv
- H. Bao, L. Dong, S. Piao and F. Wei, “BEiT: BERT Pre-Training of Image Transformers,” ICLR, 2022. arXiv
- M. Caron et al., “Emerging Properties in Self-Supervised Vision Transformers,” ICCV, 2021. arXiv
- C. Jia et al., “Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision,” ICML, 2021. arXiv
- J. Li, D. Li, S. Savarese and S. Hoi, “BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models,” ICML, 2023. PMLR
- J.-B. Alayrac et al., “Flamingo: a Visual Language Model for Few-Shot Learning,” NeurIPS, 2022. arXiv
- H. Liu, C. Li, Y. Li and Y. J. Lee, “Improved Baselines with Visual Instruction Tuning,” CVPR, 2024. arXiv
- N. Houlsby et al., “Parameter-Efficient Transfer Learning for NLP,” ICML, 2019. PMLR
- M. Jia et al., “Visual Prompt Tuning,” ECCV, 2022. arXiv
- T. Dettmers, A. Pagnoni, A. Holtzman and L. Zettlemoyer, “QLoRA: Efficient Finetuning of Quantized LLMs,” NeurIPS, 2023. arXiv
- A. Kumar, A. Raghunathan, R. Jones, T. Ma and P. Liang, “Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution,” ICLR, 2022. arXiv
- M. Wortsman et al., “Robust Fine-Tuning of Zero-Shot Models,” CVPR, 2022. arXiv
- C. Schuhmann et al., “LAION-5B: An Open Large-Scale Dataset for Training Next Generation Image-Text Models,” NeurIPS Datasets and Benchmarks Track, 2022. arXiv
- A. Birhane, V. U. Prabhu and E. Kahembwe, “Multimodal Datasets: Misogyny, Pornography, and Malignant Stereotypes,” arXiv:2110.01963, 2021. arXiv
- P. Mayilvahanan, T. Wiedemer, E. Rusak, M. Bethge and W. Brendel, “Does CLIP’s Generalization Performance Mainly Stem from High Train-Test Similarity?,” ICLR, 2024. arXiv
- Y. Li et al., “Evaluating Object Hallucination in Large Vision-Language Models,” EMNLP, 2023. arXiv
- X. Yue et al., “MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI,” CVPR, 2024. arXiv
- M. Awais et al., “Foundational Models Defining a New Era in Vision: A Survey and Outlook,” arXiv:2307.13721, 2023. arXiv
- S. Yin et al., “A Survey on Multimodal Large Language Models,” National Science Review, 2024. arXiv
- J. Zhang, J. Huang, S. Jin and S. Lu, “Vision-Language Models for Vision Tasks: A Survey,” IEEE TPAMI, vol. 46, no. 8, 2024. arXiv
- L. Xu et al., “Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment,” arXiv:2312.12148, 2023. arXiv
- J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li and L. Fei-Fei, “ImageNet: A Large-Scale Hierarchical Image Database,” CVPR, 2009, pp. 248–255. doi
- J. Devlin, M.-W. Chang, K. Lee and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” NAACL-HLT, 2019, pp. 4171–4186. doi
- OpenAI, “GPT-4 Technical Report,” arXiv:2303.08774, 2023. arXiv