Full search page →

CLIP — from paired captions to a zero-shot classifier

Optional · paper guide 1 of 1 for ML 10 · Foundation Models and Vision–Language Learning. Citation numbers refer to the reference list at the bottom of this page.

Navigate: this page and this track: Navigate: this page and this track

Question and thesis

Can natural-language supervision make an image encoder usable on new classification tasks without fitting a new task-specific classifier? Radford et al. train image and text encoders on paired web data, then use text descriptions to construct class representations [3]. Read contrastive learning and cosine similarity first. Zero-shot refers to downstream transfer, not to an absence of supervision during pretraining.

Method and figure

The three stages train paired embeddings, construct text-based class vectors, and compare an image with those vectors. The downstream class list changes the prediction task.
Source: Radford et al., ICML 2021, PMLR 139 proceedings. Figure 1, PDF p. 2. The three stages train paired embeddings, construct text-based class vectors, and compare an image with those vectors. The downstream class list changes the prediction task. [3]

Figure 1 separates contrastive pretraining, classifier construction and zero-shot prediction. For a batch of matched pairs, similarities are

sij=exp⁡(a) ui⊤vj.s_{ij}=\exp(a)\,u_i^\top v_j.

Here uiu_i is the normalized image embedding for example ii, vjv_j is the normalized text embedding for caption jj, and aa is a learned log scale. A symmetric cross-entropy objective identifies the matched pair along each row and each column of the similarity matrix. Off-diagonal pairings serve as alternatives within the batch.

For classification, encode prompted class names with the text encoder and compare their vectors with the image vector. The class list defines the candidate set. A textual template supplies context; averaging representations from multiple templates is a separate evaluation choice. Neither operation trains a new image classifier from labeled examples of that downstream task.

Evidence

Figure 4 compares zero-shot CLIP with supervised logistic regression on frozen ResNet-50 features across 27 datasets. CLIP wins on 16 of those comparisons, while performance varies substantially by dataset. This is not a claim that it beats every fully fine-tuned specialist. The paper also studies prompt engineering and ensembling, showing why the exact text interface belongs in the evaluation protocol rather than being treated as invisible formatting.

Design analysis

Paired text supplies a different positive-pair definition from SimCLR’s two augmented images. It can describe concepts without requiring one fixed class taxonomy, but web captions are incomplete and noisy descriptions of image content. At inference, changing candidate labels or their wording changes the comparison. A similarity ranking is therefore conditional on the supplied text interface, not an intrinsic probability that a visual concept is true.

Reading checkpoint

Identify the positive entries in a batch similarity matrix. Then contrast two evaluations: zero-shot text prototypes and a classifier trained on labeled frozen image features. Which parameters or decision rules are fitted using downstream labels in each case? Why should their scores be labeled separately?

Limits and next step

Pretraining coverage, data overlap, language choices and social biases affect transfer. Prompt selection can also incorporate human task knowledge even without gradient updates. Similarities are not automatically calibrated confidence scores or reliable out-of-distribution alarms. Continue to ML 11 and the calibration guide before interpreting a softmax value as the probability that a prediction is correct.

Continue reading: Temperature Scaling — confidence without changing the class

References

  1. R. Bommasani et al., “On the Opportunities and Risks of Foundation Models,” Stanford CRFM report, arXiv:2108.07258, 2021; revised 2022. arXiv
  2. D. Caffagni et al., “The Revolution of Multimodal Large Language Models: A Survey,” Findings of ACL, 2024. arXiv
  3. A. Radford et al., “Learning Transferable Visual Models From Natural Language Supervision,” ICML, 2021. arXiv
  4. X. Zhai, B. Mustafa, A. Kolesnikov and L. Beyer, “Sigmoid Loss for Language Image Pre-Training,” ICCV, 2023. arXiv
  5. A. Kirillov et al., “Segment Anything,” ICCV, 2023. arXiv
  6. N. Ravi et al., “SAM 2: Segment Anything in Images and Videos,” ICLR, 2025. arXiv
  7. N. Carion et al., “SAM 3: Segment Anything with Concepts,” arXiv:2511.16719, 2025; revised March 2026. arXiv
  8. H. Liu, C. Li, Q. Wu and Y. J. Lee, “Visual Instruction Tuning,” NeurIPS, 2023. arXiv
  9. S. Bai et al., “Qwen3-VL Technical Report,” arXiv:2511.21631, 2025. arXiv
  10. E. J. Hu et al., “LoRA: Low-Rank Adaptation of Large Language Models,” ICLR, 2022. arXiv
  11. M. Assran et al., “V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning,” arXiv:2506.09985, 2025. arXiv
  12. J. Wang et al., “VGGT: Visual Geometry Grounded Transformer,” CVPR, 2025. arXiv
  13. L. Han et al., “A Survey of Generative Categories and Techniques in Multimodal Generative Models,” arXiv:2506.10016, 2025. arXiv
  14. C. M. Bishop and H. Bishop, Deep Learning: Foundations and Concepts, Springer, 2024. Book
  15. M. Oquab et al., “DINOv2: Learning Robust Visual Features without Supervision,” Transactions on Machine Learning Research, 2024. arXiv
  16. O. Siméoni et al., “DINOv3,” arXiv:2508.10104, 2025. arXiv
  17. J. Kaplan et al., “Scaling Laws for Neural Language Models,” arXiv:2001.08361, 2020. arXiv
  18. J. Hoffmann et al., “Training Compute-Optimal Large Language Models,” arXiv:2203.15556, 2022; published as “An empirical analysis of compute-optimal large language model training,” NeurIPS, 2022. arXiv
  19. X. Zhai, A. Kolesnikov, N. Houlsby and L. Beyer, “Scaling Vision Transformers,” CVPR, 2022. arXiv
  20. M. Cherti et al., “Reproducible Scaling Laws for Contrastive Language-Image Learning,” CVPR, 2023. arXiv
  21. A. Dosovitskiy et al., “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale,” ICLR, 2021. arXiv
  22. K. He, X. Chen, S. Xie, Y. Li, P. Dollár and R. Girshick, “Masked Autoencoders Are Scalable Vision Learners,” CVPR, 2022. arXiv
  23. H. Bao, L. Dong, S. Piao and F. Wei, “BEiT: BERT Pre-Training of Image Transformers,” ICLR, 2022. arXiv
  24. M. Caron et al., “Emerging Properties in Self-Supervised Vision Transformers,” ICCV, 2021. arXiv
  25. C. Jia et al., “Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision,” ICML, 2021. arXiv
  26. J. Li, D. Li, S. Savarese and S. Hoi, “BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models,” ICML, 2023. PMLR
  27. J.-B. Alayrac et al., “Flamingo: a Visual Language Model for Few-Shot Learning,” NeurIPS, 2022. arXiv
  28. H. Liu, C. Li, Y. Li and Y. J. Lee, “Improved Baselines with Visual Instruction Tuning,” CVPR, 2024. arXiv
  29. N. Houlsby et al., “Parameter-Efficient Transfer Learning for NLP,” ICML, 2019. PMLR
  30. M. Jia et al., “Visual Prompt Tuning,” ECCV, 2022. arXiv
  31. T. Dettmers, A. Pagnoni, A. Holtzman and L. Zettlemoyer, “QLoRA: Efficient Finetuning of Quantized LLMs,” NeurIPS, 2023. arXiv
  32. A. Kumar, A. Raghunathan, R. Jones, T. Ma and P. Liang, “Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution,” ICLR, 2022. arXiv
  33. M. Wortsman et al., “Robust Fine-Tuning of Zero-Shot Models,” CVPR, 2022. arXiv
  34. C. Schuhmann et al., “LAION-5B: An Open Large-Scale Dataset for Training Next Generation Image-Text Models,” NeurIPS Datasets and Benchmarks Track, 2022. arXiv
  35. A. Birhane, V. U. Prabhu and E. Kahembwe, “Multimodal Datasets: Misogyny, Pornography, and Malignant Stereotypes,” arXiv:2110.01963, 2021. arXiv
  36. P. Mayilvahanan, T. Wiedemer, E. Rusak, M. Bethge and W. Brendel, “Does CLIP’s Generalization Performance Mainly Stem from High Train-Test Similarity?,” ICLR, 2024. arXiv
  37. Y. Li et al., “Evaluating Object Hallucination in Large Vision-Language Models,” EMNLP, 2023. arXiv
  38. X. Yue et al., “MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI,” CVPR, 2024. arXiv
  39. M. Awais et al., “Foundational Models Defining a New Era in Vision: A Survey and Outlook,” arXiv:2307.13721, 2023. arXiv
  40. S. Yin et al., “A Survey on Multimodal Large Language Models,” National Science Review, 2024. arXiv
  41. J. Zhang, J. Huang, S. Jin and S. Lu, “Vision-Language Models for Vision Tasks: A Survey,” IEEE TPAMI, vol. 46, no. 8, 2024. arXiv
  42. L. Xu et al., “Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment,” arXiv:2312.12148, 2023. arXiv
  43. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li and L. Fei-Fei, “ImageNet: A Large-Scale Hierarchical Image Database,” CVPR, 2009, pp. 248–255. doi
  44. J. Devlin, M.-W. Chang, K. Lee and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” NAACL-HLT, 2019, pp. 4171–4186. doi
  45. OpenAI, “GPT-4 Technical Report,” arXiv:2303.08774, 2023. arXiv