CLIP — from paired captions to a zero-shot classifier
Optional · paper guide 1 of 1 for ML 10 · Foundation Models and Vision–Language Learning. Citation numbers refer to the reference list at the bottom of this page.
Navigate: this page and this track: Navigate: this page and this track
Question and thesis
Can natural-language supervision make an image encoder usable on new classification tasks without fitting a new task-specific classifier? Radford et al. train image and text encoders on paired web data, then use text descriptions to construct class representations [3]. Read contrastive learning and cosine similarity first. Zero-shot refers to downstream transfer, not to an absence of supervision during pretraining.
Method and figure

Figure 1 separates contrastive pretraining, classifier construction and zero-shot prediction. For a batch of matched pairs, similarities are
Here is the normalized image embedding for example , is the normalized text embedding for caption , and is a learned log scale. A symmetric cross-entropy objective identifies the matched pair along each row and each column of the similarity matrix. Off-diagonal pairings serve as alternatives within the batch.
For classification, encode prompted class names with the text encoder and compare their vectors with the image vector. The class list defines the candidate set. A textual template supplies context; averaging representations from multiple templates is a separate evaluation choice. Neither operation trains a new image classifier from labeled examples of that downstream task.
Evidence
Figure 4 compares zero-shot CLIP with supervised logistic regression on frozen ResNet-50 features across 27 datasets. CLIP wins on 16 of those comparisons, while performance varies substantially by dataset. This is not a claim that it beats every fully fine-tuned specialist. The paper also studies prompt engineering and ensembling, showing why the exact text interface belongs in the evaluation protocol rather than being treated as invisible formatting.
Design analysis
Paired text supplies a different positive-pair definition from SimCLR’s two augmented images. It can describe concepts without requiring one fixed class taxonomy, but web captions are incomplete and noisy descriptions of image content. At inference, changing candidate labels or their wording changes the comparison. A similarity ranking is therefore conditional on the supplied text interface, not an intrinsic probability that a visual concept is true.
Reading checkpoint
Identify the positive entries in a batch similarity matrix. Then contrast two evaluations: zero-shot text prototypes and a classifier trained on labeled frozen image features. Which parameters or decision rules are fitted using downstream labels in each case? Why should their scores be labeled separately?
Limits and next step
Pretraining coverage, data overlap, language choices and social biases affect transfer. Prompt selection can also incorporate human task knowledge even without gradient updates. Similarities are not automatically calibrated confidence scores or reliable out-of-distribution alarms. Continue to ML 11 and the calibration guide before interpreting a softmax value as the probability that a prediction is correct.
Continue reading: Temperature Scaling — confidence without changing the class
References
- R. Bommasani et al., “On the Opportunities and Risks of Foundation Models,” Stanford CRFM report, arXiv:2108.07258, 2021; revised 2022. arXiv
- D. Caffagni et al., “The Revolution of Multimodal Large Language Models: A Survey,” Findings of ACL, 2024. arXiv
- A. Radford et al., “Learning Transferable Visual Models From Natural Language Supervision,” ICML, 2021. arXiv
- X. Zhai, B. Mustafa, A. Kolesnikov and L. Beyer, “Sigmoid Loss for Language Image Pre-Training,” ICCV, 2023. arXiv
- A. Kirillov et al., “Segment Anything,” ICCV, 2023. arXiv
- N. Ravi et al., “SAM 2: Segment Anything in Images and Videos,” ICLR, 2025. arXiv
- N. Carion et al., “SAM 3: Segment Anything with Concepts,” arXiv:2511.16719, 2025; revised March 2026. arXiv
- H. Liu, C. Li, Q. Wu and Y. J. Lee, “Visual Instruction Tuning,” NeurIPS, 2023. arXiv
- S. Bai et al., “Qwen3-VL Technical Report,” arXiv:2511.21631, 2025. arXiv
- E. J. Hu et al., “LoRA: Low-Rank Adaptation of Large Language Models,” ICLR, 2022. arXiv
- M. Assran et al., “V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning,” arXiv:2506.09985, 2025. arXiv
- J. Wang et al., “VGGT: Visual Geometry Grounded Transformer,” CVPR, 2025. arXiv
- L. Han et al., “A Survey of Generative Categories and Techniques in Multimodal Generative Models,” arXiv:2506.10016, 2025. arXiv
- C. M. Bishop and H. Bishop, Deep Learning: Foundations and Concepts, Springer, 2024. Book
- M. Oquab et al., “DINOv2: Learning Robust Visual Features without Supervision,” Transactions on Machine Learning Research, 2024. arXiv
- O. Siméoni et al., “DINOv3,” arXiv:2508.10104, 2025. arXiv
- J. Kaplan et al., “Scaling Laws for Neural Language Models,” arXiv:2001.08361, 2020. arXiv
- J. Hoffmann et al., “Training Compute-Optimal Large Language Models,” arXiv:2203.15556, 2022; published as “An empirical analysis of compute-optimal large language model training,” NeurIPS, 2022. arXiv
- X. Zhai, A. Kolesnikov, N. Houlsby and L. Beyer, “Scaling Vision Transformers,” CVPR, 2022. arXiv
- M. Cherti et al., “Reproducible Scaling Laws for Contrastive Language-Image Learning,” CVPR, 2023. arXiv
- A. Dosovitskiy et al., “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale,” ICLR, 2021. arXiv
- K. He, X. Chen, S. Xie, Y. Li, P. Dollár and R. Girshick, “Masked Autoencoders Are Scalable Vision Learners,” CVPR, 2022. arXiv
- H. Bao, L. Dong, S. Piao and F. Wei, “BEiT: BERT Pre-Training of Image Transformers,” ICLR, 2022. arXiv
- M. Caron et al., “Emerging Properties in Self-Supervised Vision Transformers,” ICCV, 2021. arXiv
- C. Jia et al., “Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision,” ICML, 2021. arXiv
- J. Li, D. Li, S. Savarese and S. Hoi, “BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models,” ICML, 2023. PMLR
- J.-B. Alayrac et al., “Flamingo: a Visual Language Model for Few-Shot Learning,” NeurIPS, 2022. arXiv
- H. Liu, C. Li, Y. Li and Y. J. Lee, “Improved Baselines with Visual Instruction Tuning,” CVPR, 2024. arXiv
- N. Houlsby et al., “Parameter-Efficient Transfer Learning for NLP,” ICML, 2019. PMLR
- M. Jia et al., “Visual Prompt Tuning,” ECCV, 2022. arXiv
- T. Dettmers, A. Pagnoni, A. Holtzman and L. Zettlemoyer, “QLoRA: Efficient Finetuning of Quantized LLMs,” NeurIPS, 2023. arXiv
- A. Kumar, A. Raghunathan, R. Jones, T. Ma and P. Liang, “Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution,” ICLR, 2022. arXiv
- M. Wortsman et al., “Robust Fine-Tuning of Zero-Shot Models,” CVPR, 2022. arXiv
- C. Schuhmann et al., “LAION-5B: An Open Large-Scale Dataset for Training Next Generation Image-Text Models,” NeurIPS Datasets and Benchmarks Track, 2022. arXiv
- A. Birhane, V. U. Prabhu and E. Kahembwe, “Multimodal Datasets: Misogyny, Pornography, and Malignant Stereotypes,” arXiv:2110.01963, 2021. arXiv
- P. Mayilvahanan, T. Wiedemer, E. Rusak, M. Bethge and W. Brendel, “Does CLIP’s Generalization Performance Mainly Stem from High Train-Test Similarity?,” ICLR, 2024. arXiv
- Y. Li et al., “Evaluating Object Hallucination in Large Vision-Language Models,” EMNLP, 2023. arXiv
- X. Yue et al., “MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI,” CVPR, 2024. arXiv
- M. Awais et al., “Foundational Models Defining a New Era in Vision: A Survey and Outlook,” arXiv:2307.13721, 2023. arXiv
- S. Yin et al., “A Survey on Multimodal Large Language Models,” National Science Review, 2024. arXiv
- J. Zhang, J. Huang, S. Jin and S. Lu, “Vision-Language Models for Vision Tasks: A Survey,” IEEE TPAMI, vol. 46, no. 8, 2024. arXiv
- L. Xu et al., “Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment,” arXiv:2312.12148, 2023. arXiv
- J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li and L. Fei-Fei, “ImageNet: A Large-Scale Hierarchical Image Database,” CVPR, 2009, pp. 248–255. doi
- J. Devlin, M.-W. Chang, K. Lee and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” NAACL-HLT, 2019, pp. 4171–4186. doi
- OpenAI, “GPT-4 Technical Report,” arXiv:2303.08774, 2023. arXiv