論文導讀
選讀:ML 10・基礎模型與視覺語言學習 所依據論文的導讀,每篇一頁:說明它要解決的問題、核心想法、結果,以及閱讀時該注意什麼。
導覽:本頁與本系列: 導覽:本頁與本系列
以下為選讀導讀。先讀問題與原圖,再逐層展開證據、設計分析及閱讀檢查點。
- CLIP:從影像文字配對到零樣本分類器
自然語言監督能否讓影像編碼器適用於新分類任務,而不必重新擬合任務專用分類器?
參考文獻
- R. Bommasani et al., “On the Opportunities and Risks of Foundation Models,” Stanford CRFM report, arXiv:2108.07258, 2021; revised 2022. arXiv
- D. Caffagni et al., “The Revolution of Multimodal Large Language Models: A Survey,” Findings of ACL, 2024. arXiv
- A. Radford et al., “Learning Transferable Visual Models From Natural Language Supervision,” ICML, 2021. arXiv
- X. Zhai, B. Mustafa, A. Kolesnikov and L. Beyer, “Sigmoid Loss for Language Image Pre-Training,” ICCV, 2023. arXiv
- A. Kirillov et al., “Segment Anything,” ICCV, 2023. arXiv
- N. Ravi et al., “SAM 2: Segment Anything in Images and Videos,” ICLR, 2025. arXiv
- N. Carion et al., “SAM 3: Segment Anything with Concepts,” arXiv:2511.16719, 2025; revised March 2026. arXiv
- H. Liu, C. Li, Q. Wu and Y. J. Lee, “Visual Instruction Tuning,” NeurIPS, 2023. arXiv
- S. Bai et al., “Qwen3-VL Technical Report,” arXiv:2511.21631, 2025. arXiv
- E. J. Hu et al., “LoRA: Low-Rank Adaptation of Large Language Models,” ICLR, 2022. arXiv
- M. Assran et al., “V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning,” arXiv:2506.09985, 2025. arXiv
- J. Wang et al., “VGGT: Visual Geometry Grounded Transformer,” CVPR, 2025. arXiv
- L. Han et al., “A Survey of Generative Categories and Techniques in Multimodal Generative Models,” arXiv:2506.10016, 2025. arXiv
- C. M. Bishop and H. Bishop, Deep Learning: Foundations and Concepts, Springer, 2024. Book
- M. Oquab et al., “DINOv2: Learning Robust Visual Features without Supervision,” Transactions on Machine Learning Research, 2024. arXiv
- O. Siméoni et al., “DINOv3,” arXiv:2508.10104, 2025. arXiv
- J. Kaplan et al., “Scaling Laws for Neural Language Models,” arXiv:2001.08361, 2020. arXiv
- J. Hoffmann et al., “Training Compute-Optimal Large Language Models,” arXiv:2203.15556, 2022; published as “An empirical analysis of compute-optimal large language model training,” NeurIPS, 2022. arXiv
- X. Zhai, A. Kolesnikov, N. Houlsby and L. Beyer, “Scaling Vision Transformers,” CVPR, 2022. arXiv
- M. Cherti et al., “Reproducible Scaling Laws for Contrastive Language-Image Learning,” CVPR, 2023. arXiv
- A. Dosovitskiy et al., “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale,” ICLR, 2021. arXiv
- K. He, X. Chen, S. Xie, Y. Li, P. Dollár and R. Girshick, “Masked Autoencoders Are Scalable Vision Learners,” CVPR, 2022. arXiv
- H. Bao, L. Dong, S. Piao and F. Wei, “BEiT: BERT Pre-Training of Image Transformers,” ICLR, 2022. arXiv
- M. Caron et al., “Emerging Properties in Self-Supervised Vision Transformers,” ICCV, 2021. arXiv
- C. Jia et al., “Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision,” ICML, 2021. arXiv
- J. Li, D. Li, S. Savarese and S. Hoi, “BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models,” ICML, 2023. PMLR
- J.-B. Alayrac et al., “Flamingo: a Visual Language Model for Few-Shot Learning,” NeurIPS, 2022. arXiv
- H. Liu, C. Li, Y. Li and Y. J. Lee, “Improved Baselines with Visual Instruction Tuning,” CVPR, 2024. arXiv
- N. Houlsby et al., “Parameter-Efficient Transfer Learning for NLP,” ICML, 2019. PMLR
- M. Jia et al., “Visual Prompt Tuning,” ECCV, 2022. arXiv
- T. Dettmers, A. Pagnoni, A. Holtzman and L. Zettlemoyer, “QLoRA: Efficient Finetuning of Quantized LLMs,” NeurIPS, 2023. arXiv
- A. Kumar, A. Raghunathan, R. Jones, T. Ma and P. Liang, “Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution,” ICLR, 2022. arXiv
- M. Wortsman et al., “Robust Fine-Tuning of Zero-Shot Models,” CVPR, 2022. arXiv
- C. Schuhmann et al., “LAION-5B: An Open Large-Scale Dataset for Training Next Generation Image-Text Models,” NeurIPS Datasets and Benchmarks Track, 2022. arXiv
- A. Birhane, V. U. Prabhu and E. Kahembwe, “Multimodal Datasets: Misogyny, Pornography, and Malignant Stereotypes,” arXiv:2110.01963, 2021. arXiv
- P. Mayilvahanan, T. Wiedemer, E. Rusak, M. Bethge and W. Brendel, “Does CLIP’s Generalization Performance Mainly Stem from High Train-Test Similarity?,” ICLR, 2024. arXiv
- Y. Li et al., “Evaluating Object Hallucination in Large Vision-Language Models,” EMNLP, 2023. arXiv
- X. Yue et al., “MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI,” CVPR, 2024. arXiv
- M. Awais et al., “Foundational Models Defining a New Era in Vision: A Survey and Outlook,” arXiv:2307.13721, 2023. arXiv
- S. Yin et al., “A Survey on Multimodal Large Language Models,” National Science Review, 2024. arXiv
- J. Zhang, J. Huang, S. Jin and S. Lu, “Vision-Language Models for Vision Tasks: A Survey,” IEEE TPAMI, vol. 46, no. 8, 2024. arXiv
- L. Xu et al., “Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment,” arXiv:2312.12148, 2023. arXiv
- J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li and L. Fei-Fei, “ImageNet: A Large-Scale Hierarchical Image Database,” CVPR, 2009, pp. 248–255. doi
- J. Devlin, M.-W. Chang, K. Lee and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” NAACL-HLT, 2019, pp. 4171–4186. doi
- OpenAI, “GPT-4 Technical Report,” arXiv:2303.08774, 2023. arXiv