Full search page →

ML 10 · Foundation Models and Vision–Language Learning

Machine LearningAdvanced27 minUpdated Nov 6, 2026

Needs: ML 06 · Representation Learning and Self-Supervision, ML 09 · Autoregressive Vision: Pixels, Tokens and Scales, ML 08 · Diffusion Models and Flow Matching

Navigate: this page and this track: Navigate: this page and this track
Explore this chapter 1 interactive experiment

Move paired and unpaired unit vectors, then inspect the normalized matching loss used by contrastive learning.

What you’ll learn

  • Define foundation models through pretraining and reuse.
  • Interpret scaling laws within their measured regimes.
  • Compare supervision sources and reusable vision capabilities.
  • Trace image–text alignment, visual prompts and multimodal connectors.
  • Choose adaptation methods and test task performance, robustness and limitations.

The big picture

For most of the history of computer vision, each task had its own model: one network for classifying flowers, another for detecting cars, another for segmenting tumors, each trained from scratch or from an ImageNet [43] classifier. A foundation model is trained once, on broad data and at large scale, and then adapted to many downstream tasks [1]. The word describes how the model is trained and reused, not a new layer type. A DINOv2 [15] or DINOv3 [16] image encoder, a CLIP-style image–text model [3], a promptable segmenter such as SAM [5], a multimodal language model and a broadly pretrained image generator can all play this role.

Why does this matter? Three reasons. First, labels are expensive and pretraining moves most of the learning to data that needs no hand labels (captions, the images themselves). Second, one backbone serves many tasks, which concentrates engineering effort. Third, it changes the interface: instead of retraining, you can often specify a task with a text prompt, a click or a few examples. Bommasani et al. call the resulting concentration of methods homogenization, and warn that it also concentrates failure: a flaw in one shared backbone is inherited by every system built on it [1].

This chapter builds on ML 06 · Representation Learning (what makes a feature reusable), ML 05 · Vision Transformers (the usual backbone) and the generative chapters ML 08 and ML 09. The textbook treatment of transfer learning and representation in Bishop and Bishop [14] is a good companion. We follow the architectural map of the vision foundation-model survey by Awais et al. [39] and the multimodal survey by Caffagni et al. [2], with representative reports verified for this October 2026 edition. Examples illustrate families; they do not rank commercial products or claim that one checkpoint is best everywhere.

What makes a model a “foundation” model

Plain version. A foundation model is a large model pretrained on broad data with a generic objective, so that its internal representations or its interface can be reused for tasks it was never specifically trained on.

Bommasani et al. [1] coined the term in 2021 and emphasized two properties:

  • Emergence. Capabilities arise from scale and the training objective rather than being designed explicitly. For example, CLIP was trained only to match images with captions, yet it can classify images into categories named in text [3]; DINO was trained without any labels, yet its attention maps outline objects [24].
  • Homogenization. Many applications converge on a small number of pretrained backbones and recipes.

Be careful with “emergence”. In this chapter it means only a capability that was not a training target. It does not mean an unpredictable or mysterious jump; whether a capability appears suddenly often depends on the metric used to measure it.

Formally, let fθf_\theta be a model pretrained on an upstream dataset Dup\mathcal D_{\mathrm{up}} with objective Lpre\mathcal L_{\mathrm{pre}}:

θ⋆=arg⁡min⁡θ Ex∼Dup[Lpre(fθ;x)].\theta^\star=\arg\min_\theta\ \mathbb E_{x\sim\mathcal D_{\mathrm{up}}}\big[\mathcal L_{\mathrm{pre}}(f_\theta;x)\big].

A downstream task kk supplies a (usually small) dataset Dk\mathcal D_k and a task loss ℓk\ell_k. Adaptation chooses a small set of new or changed parameters ϕk\phi_k and solves

ϕk⋆=arg⁡min⁡ϕk ∑(x,y)∈Dkℓk(hϕk(fθ⋆,x),y),\phi_k^\star=\arg\min_{\phi_k}\ \sum_{(x,y)\in\mathcal D_k}\ell_k\big(h_{\phi_k}(f_{\theta^\star},x),y\big),

where hϕkh_{\phi_k} is whatever we put on top of or inside the pretrained model: a linear head, an adapter, a LoRA update, a prompt, or nothing at all (pure prompting, ϕk=∅\phi_k=\varnothing). The whole chapter is about the two lines above: what Lpre\mathcal L_{\mathrm{pre}} and Dup\mathcal D_{\mathrm{up}} should be, and how small ϕk\phi_k can be.

A foundation model does not have to produce text. A frozen image encoder whose features feed a hundred different classifiers is a foundation model. Conversely, a large model trained for one narrow task is not one, however big it is.

Scaling laws

Plain version. Scaling laws fit how loss changes with model size, data and compute. They help plan runs within comparable regimes, but do not guarantee extrapolated gains.

Kaplan et al. [17] trained families of Transformer language models and reported that the loss scales as a power law in model size, dataset size and compute, with some trends spanning more than seven orders of magnitude, while width and depth mattered little within the tested range. A power law in model size NN reads

L(N)=(NcN)αN,L(N)=\left(\frac{N_c}{N}\right)^{\alpha_N},

where LL is held-out cross-entropy loss, NN the number of non-embedding parameters, NcN_c a fitted constant with the same units as NN, and αN>0\alpha_N>0 the fitted exponent. Taking logs gives log⁡L=αNlog⁡Nc−αNlog⁡N\log L=\alpha_N\log N_c-\alpha_N\log N: a straight line with slope −αN-\alpha_N.

Real curves bend because some loss can never be removed (the data itself is noisy or ambiguous). Hoffmann et al. [18] therefore fitted a form with an irreducible term:

L(N,D)=E+ANα+BDβ,L(N,D)=E+\frac{A}{N^{\alpha}}+\frac{B}{D^{\beta}},

where DD is the number of training tokens, EE the irreducible loss, and A,B,α,βA,B,\alpha,\beta fitted constants. For dense Transformers a common approximation of training compute is C≈6NDC\approx 6ND floating-point operations [17] (roughly two for the forward and four for the backward pass, per parameter per token). Fixing CC and minimizing LL over NN (with D=C/6ND=C/6N) gives a compute-optimal model size. Hoffmann et al. trained over 400 language models and concluded that, for compute-optimal training, model size and training tokens should be scaled equally — every doubling of model size should come with a doubling of tokens [18]. That differed from Kaplan et al.’s earlier recommendation of very large models trained on relatively modest data [17]; the disagreement is a useful reminder that fitted exponents depend on the experimental protocol (learning-rate schedules, which runs are included, what is counted as compute).

Left: synthetic loss points versus compute on log-log axes, with a fitted curve that bends toward a horizontal floor and a dotted pure power-law line that keeps falling. Right: three U-shaped curves of predicted loss versus model size at three fixed compute budgets, each with its minimum marked.
Figure 1 — (a) A power law with an irreducible floor E: a pure power law (dotted) keeps falling and over-promises at large scale. (b) With compute fixed, too small a model under-fits and too large a model sees too few tokens, so each budget has an optimal size. All constants are synthetic.

Vision has its own scaling studies. Zhai et al. [19] characterized the relationship between error rate, data and compute for Vision Transformers [21] trained on large labeled datasets, scaling to two-billion-parameter models. Cherti et al. [20] trained CLIP-style models on the public LAION dataset [34] and found power-law scaling for zero-shot classification, retrieval, linear probing and fine-tuning — but also that OpenAI’s CLIP and OpenCLIP, with the same architectures and similar recipes, scale differently because their training distributions differ. Data is not a scalar.

Code: fit a scaling law and extrapolate

The snippet fits L(C)=E+A C−αL(C)=E+A\,C^{-\alpha} to the cheaper of 13 synthetic training runs and then predicts the expensive ones. Everything is synthetic; the point is the procedure.

Code · Python · 22 lines
import numpy as np
from scipy.optimize import curve_fit

rng = np.random.default_rng(0)
# Synthetic "runs": loss = E + A * C^(-alpha), with 2% multiplicative noise.
E_true, A_true, alpha_true = 1.7, 40.0, 0.3
compute = np.logspace(0, 6, 13)            # arbitrary compute units
loss = (E_true + A_true * compute**-alpha_true) * np.exp(.02 * rng.normal(size=13))

def law(c, E, A, alpha):
    return E + A * c**-alpha

# Fit only the cheaper runs, then extrapolate to the larger ones.
small = compute <= 1e4
(E, A, alpha), _ = curve_fit(law, compute[small], loss[small], p0=(1, 10, .5))
print(f"fit: E={E:.2f}  A={A:.1f}  alpha={alpha:.3f}")
for c, l in zip(compute[~small], loss[~small]):
    print(f"C={c:9.0f}  observed {l:.3f}  predicted {law(c, E, A, alpha):.3f}")

# Naive straight line in log-log space ignores the floor E.
slope = np.polyfit(np.log(compute[small]), np.log(loss[small]), 1)[0]
print(f"pure power-law slope without floor: {slope:.3f}")

Output: the fit recovers E≈1.73E\approx1.73, A≈40.0A\approx40.0, α≈0.300\alpha\approx0.300; the extrapolated losses are within about 7% of the noisy observations; and a straight log–log line without the floor gives a slope of about −0.25-0.25, a wrong exponent. Two lessons transfer to real practice: always include the floor, and treat extrapolation over many orders of magnitude as a forecast with error bars.

What scaling laws do not tell you

  • They predict average loss on the pretraining distribution, not accuracy on your task. Downstream metrics may improve smoothly, saturate, or move in steps.
  • They hold for a fixed recipe and data distribution [20]. Better data filtering or a different objective changes the constants.
  • They say nothing about inference cost. A smaller model trained on more data can be cheaper to deploy at the same loss, which is why many deployed models are trained beyond the compute-optimal point.
  • Data is finite. If the high-quality data runs out, the B/DβB/D^{\beta} term stops shrinking however much compute you add.

Pre-training objectives: where does the target come from?

Plain version. Every pretraining method invents a prediction task. The difference is where the “answer” comes from: a human label, a caption, a hidden part of the same image, another network, or the next token.

A tree diagram: the question 'where does the target come from?' branches into five objective families — supervised classification, contrastive image-text, masked image modeling, self-distillation and generative next-token — each with its source of targets.
Figure 2 — Five families of pre-training objectives, organized by the source of the training target.
ObjectiveTarget comes fromRepresentative modelsTypical strength
Supervised classificationHuman labelsViT on large labeled sets [19], [21]Strong global semantics for the label space
Contrastive image–textPaired captions / alt textCLIP [3], ALIGN [25], SigLIP [4]Text-defined recognition, retrieval
Masked image modelingHidden patches of the same imageBEiT [23], MAE [22]Strong after fine-tuning; local detail
Self-distillationA teacher network’s output on another viewDINO [24], DINOv2 [15], DINOv3 [16]Frozen features, dense correspondence
Generative / next-tokenNext token or clean signalLLMs, diffusion (ML 07), AR (ML 08)Generation; language interface

Supervised pre-training

Train a classifier with cross-entropy on a large labeled dataset, then drop the final layer. This was the ImageNet recipe for a decade and still works at scale [19], [21]. Its weakness is that the features are shaped by a fixed label set: distinctions that the labels do not require may be discarded.

Contrastive image–text learning

Captions come for free with web images, so CLIP [3] and ALIGN [25] train an image encoder and a text encoder together so that matching pairs score higher than mismatched ones within a batch. ALIGN showed that a large, noisy alt-text corpus with minimal filtering can still give strong representations [25]. The loss is detailed in the next section.

Masked image modeling

Hide most of an image and predict what is missing. BEiT [23] masks patches and predicts discrete visual tokens for them, borrowing the masked-language-modeling idea from BERT [44]. MAE [22] predicts the raw pixels directly and observes that masking a high proportion, e.g. 75% of patches, creates a nontrivial task; its encoder sees only the visible patches and a lightweight decoder reconstructs the rest, which makes training efficient. With M\mathcal M the set of masked patch indices, xpx_p the (normalized) pixels of patch pp and x^p\hat x_p the reconstruction,

LMAE=1∣M∣∑p∈M∥x^p−xp∥22.\mathcal L_{\mathrm{MAE}}=\frac1{|\mathcal M|}\sum_{p\in\mathcal M}\|\hat x_p-x_p\|_2^2 .

The loss is computed only on masked patches: predicting the visible ones would be trivial copying. Masked-modeling features are often excellent after fine-tuning but weaker as frozen features for linear probing than contrastive or self-distilled ones.

Self-distillation: the DINO family

DINO [24] — described by its authors as self-distillation with no labels — trains a student network gsg_s to match a teacher network gtg_t on different augmented views (crops) of the same image. Both output KK-dimensional scores; probabilities are softmaxes with temperatures τs\tau_s and τt\tau_t:

Ps(x′)=softmax ⁣(gs(x′)τs),Pt(x)=softmax ⁣(gt(x)−cτt),L=−∑k=1KPt(x)klog⁡Ps(x′)k.P_s(x')=\mathrm{softmax}\!\left(\frac{g_s(x')}{\tau_s}\right),\qquad P_t(x)=\mathrm{softmax}\!\left(\frac{g_t(x)-c}{\tau_t}\right),\qquad \mathcal L=-\sum_{k=1}^{K}P_t(x)_k\log P_s(x')_k .

Here xx and x′x' are two views, cc is a running mean of teacher outputs (centering), and a small τt\tau_t makes the teacher’s distribution sharp (sharpening). Centering stops one dimension from dominating; sharpening stops the output from becoming uniform; together they avoid collapse, the trivial solution where every image maps to the same output. The teacher receives no gradients. Its weights are an exponential moving average (the momentum encoder) of the student’s:

θt←λ θt+(1−λ) θs,0<λ<1.\theta_t\leftarrow\lambda\,\theta_t+(1-\lambda)\,\theta_s,\qquad 0\lt\lambda\lt1 .

DINO also uses multi-crop training (several small local crops for the student, global crops for the teacher), which forces local-to-global correspondence. Its authors report that the resulting ViT features contain explicit information about object segmentation that does not emerge as clearly with supervised ViTs or convnets, and that they work well as k-nearest-neighbor classifiers [24].

DINOv2 [15] scales this recipe and, crucially, curates the data: an automatic pipeline builds a dedicated, diverse, curated image dataset instead of using uncurated web images, a 1-billion-parameter ViT is trained, and it is distilled into smaller models. The authors report that these frozen features surpass OpenCLIP on most of their image-level and pixel-level benchmarks [15]. DINOv3 [16] scales data and model further and introduces Gram anchoring, a technique aimed at a known problem: dense (per-patch) feature maps degrading during long training schedules. It also adds post-hoc strategies for flexibility in resolution, model size and alignment with text.

Why dense features matter. A classifier needs one vector per image. Segmentation, depth estimation, correspondence and tracking need a good vector per patch. Self-distilled encoders are popular frozen backbones for these dense tasks precisely because their patch features stay semantically meaningful without fine-tuning.

Generative pre-training

Predicting the next token (language models, autoregressive image models in ML 09) or denoising (diffusion in ML 08) is also a pretraining objective. For language it produces the most flexible interface — instructions in natural language — which is why multimodal models attach vision to a pretrained language model rather than the other way round.

Image and text in a shared space: CLIP and SigLIP

Plain version. Align paired image and text embeddings, then score images against candidate descriptions. Text-defined classification still depends on prompts and learned coverage.

CLIP [3] normalizes image embeddings viv_i and text embeddings tjt_j to unit length and forms scores sij=vi⊤tj/τs_{ij}=v_i^\top t_j/\tau with a learned temperature τ\tau. For a batch of BB paired examples,

LCLIP=12[1B∑i−log⁡exp⁡(sii)∑jexp⁡(sij)+1B∑j−log⁡exp⁡(sjj)∑iexp⁡(sij)].\mathcal L_{\mathrm{CLIP}}= \frac12\left[ \frac1B\sum_i-\log\frac{\exp(s_{ii})}{\sum_j\exp(s_{ij})} +\frac1B\sum_j-\log\frac{\exp(s_{jj})}{\sum_i\exp(s_{ij})} \right].

Each image should select its matching caption among the BB captions (first term, a row-wise softmax), and each caption its matching image (second term, column-wise). All other pairs in the batch act as negatives. Training uses paired text supervision, usually from existing captions or alt text; “zero-shot” downstream use does not mean the model was trained without supervision.

At inference, encode candidate descriptions such as “a photo of a dog” and compare an image with them. The text embeddings replace the class weights of a learned classifier:

y^=arg⁡max⁡kv(x)⊤t(promptk).\hat y=\arg\max_k v(x)^\top t(\mathrm{prompt}_k).

The candidate set and wording affect the result; averaging the embeddings of several templates (“a photo of a kk”, “a close-up of a kk”) is a common way to reduce sensitivity. Zero-shot means no task-specific parameter training under the stated protocol; it does not guarantee that the classes or even the test images were absent from pretraining (see Risks and evaluation).

Code: paired loss and retrieval mechanics

These synthetic embeddings make the computation transparent. They are not outputs from a pretrained model, and their matches do not demonstrate real-world zero-shot accuracy.

Code · Python · 14 lines
import torch
from torch.nn import functional as F

images = F.normalize(torch.tensor([[1., .1, 0.], [.1, 1., 0.],
                                   [0., .1, 1.]]), dim=-1)
texts = F.normalize(torch.eye(3), dim=-1)
scores = images @ texts.T / .1
labels = torch.arange(3)
loss = (F.cross_entropy(scores, labels) +
        F.cross_entropy(scores.T, labels))/2
print("matching text:", scores.argmax(1).tolist())
print("paired loss:", round(loss.item(), 4))
ambiguous = F.normalize(torch.tensor([[.5, .5, .2]]), dim=-1)
print("cosines:", (ambiguous @ texts.T).tolist())

It prints matching text: [0, 1, 2], a paired loss of about 0.0002, and cosines of about 0.68, 0.68, 0.27 for the ambiguous image: it is equally close to two descriptions, and no softmax can tell you which is right.

A three-by-three matrix of illustrative image-text cosine similarities; two images match their own text with cosine near one, and an ambiguous image has cosine 0.68 with two texts.
Figure 3 — Cosine similarity ranks candidates. It is not a calibrated probability that a concept is present.

SigLIP: a sigmoid instead of a softmax

SigLIP [4] replaces the batch-normalized softmax with an independent binary decision for every image–text pair. With labels yij=+1y_{ij}=+1 for matching pairs and −1-1 otherwise, a learned scale aa and bias bb,

LSigLIP=−1B∑i=1B∑j=1Blog⁡σ ⁣(yij(a vi⊤tj+b)),\mathcal L_{\mathrm{SigLIP}}=-\frac1B\sum_{i=1}^{B}\sum_{j=1}^{B}\log\sigma\!\big(y_{ij}(a\,v_i^\top t_j+b)\big),

where σ(u)=1/(1+e−u)\sigma(u)=1/(1+e^{-u}), so each term equals log⁡(1+exp⁡(−yij(a vi⊤tj+b)))\log(1+\exp(-y_{ij}(a\,v_i^\top t_j+b))). Because no term needs a sum over the whole batch, the loss does not require a global normalization across devices, which simplifies large-batch training. Since there are BB positives and B2−BB^2-B negatives, the bias bb is initialized so that the initial predictions are strongly negative.

Shared embeddings support retrieval and open-vocabulary recognition, but fine-grained counting, negation (“a dog without a collar”), spatial relations and rare domain terms are known weak points. Evaluate those requirements directly rather than inferring them from a strong overall classification score.

Optional: CLIP (in the Paper guides). Read the original figure, evidence, design choices and limitations.

Promptable segmentation: the Segment Anything family

Plain version. A prompt tells the model which region or concept you want, and the output is a mask.

SAM [5] combines a heavy image encoder (run once per image), a light prompt encoder and a light mask decoder, so many prompts can be answered quickly on the same image. Spatial prompts — points, boxes or rough masks — select objects without a fixed class vocabulary. Because a single click is ambiguous (a click on a shirt could mean the shirt or the whole person), SAM predicts several candidate masks with confidence scores. It was trained on SA-1B, over 1 billion masks on 11 million licensed and privacy-respecting images, collected with a model-in-the-loop data engine [5].

SAM 2 [6] extends promptable segmentation to video with a streaming memory: the model stores features of past frames and of the prompted object, and propagates the mask through time; corrective clicks on later frames update the memory.

SAM 3 [7], released in 2025 with a report revised in March 2026, introduces promptable concept segmentation: given a short noun phrase (e.g. “yellow school bus”), image exemplars, or both, it returns masks and identities for all matching instances. Its image-level detector and memory-based video tracker share one backbone, and a presence head decouples recognizing whether the concept occurs at all from localizing its instances [7].

These interfaces supply different information: a spatial point says where, a noun phrase says what, an exemplar says what it looks like. Report prompt creation cost and whether prompts come from people, ground truth or an automatic detector. A “zero-shot segmentation” score obtained with ground-truth boxes as prompts hides substantial annotation information.

Multimodal large language models

Plain version. Turn visual features into “words” a language model can read, then teach it to follow visual instructions.

A common design has

image/video →vision encoder h→connector z→language decoder with text answer.\text{image/video}\ \xrightarrow{\text{vision encoder}}\ h \xrightarrow{\text{connector}}\ z \xrightarrow{\text{language decoder with text}}\ \text{answer}.

Here hh is a grid of visual features (for a ViT, one vector per patch), zz is a sequence of vectors in the language model’s token-embedding space, and the decoder treats zz like the embeddings of ordinary words.

A pipeline diagram: image or video frames go to a vision encoder, then a connector (linear, MLP, Q-Former or resampler), then into a decoder-only language model together with text prompt tokens; the model outputs answer tokens, where the loss is applied.
Figure 4 — The typical multimodal LLM pipeline. Visual features are converted into tokens in the language model's embedding space.

Connectors

  • Linear or MLP projection. LLaVA [8] uses a single projection from CLIP features to the language model’s embedding space; LLaVA-1.5 [28] reports that a simple MLP connector (with a CLIP ViT-L at 336-pixel resolution and academic VQA data) is surprisingly powerful and data-efficient. Every patch becomes one visual token.
  • Learned queries. BLIP-2 [26] inserts a lightweight Querying Transformer (Q-Former) between a frozen image encoder and a frozen language model. A fixed set of learned query vectors cross-attends to the image features, so the language model always sees a small, fixed number of visual tokens. BLIP-2 trains the Q-Former in two stages: representation learning against the frozen image encoder, then generative learning against the frozen language model.
  • Resampling and cross-attention. Flamingo [27] bridges frozen vision and language models and handles arbitrarily interleaved sequences of images and text, which enabled few-shot learning from examples written directly in the prompt.

The trade-off is token count versus detail: one token per patch keeps fine detail but makes long sequences (expensive for many images or video); a few learned queries are cheap but can lose small text.

Visual instruction tuning

LLaVA [8] used a language-only GPT-4 [45] to generate multimodal instruction-following data from image captions and boxes, then trained in two stages: first align the connector (feature alignment), then fine-tune connector and language model on the instructions. The loss is ordinary next-token cross-entropy, applied to the answer tokens only:

L=−∑i∈Alog⁡pθ(wi∣z, w<i),\mathcal L=-\sum_{i\in\mathcal A}\log p_\theta\big(w_i\mid z,\ w_{\lt i}\big),

where wiw_i are text tokens, A\mathcal A is the set of positions belonging to the assistant’s answer, and zz are the visual tokens. Instruction and prompt tokens are visible as context but masked out of the loss.

Qwen3-VL’s 2025 report [9] illustrates the current direction: a family of dense and mixture-of-experts models with natively interleaved image, video and text contexts of up to 256K tokens. It is a representative report, not evidence that every task benefits from longer contexts or more generated reasoning.

What the diagram hides

Reading a document may require preserving small text through tiling or dynamic resolution. Video questions need timestamps and temporal order. These details change token counts and preprocessing; a generic “vision encoder + LLM” diagram is only the starting point. A fluent answer can also contain objects that are not in the image (object hallucination) or wrong counts. Distinguish object grounding, OCR, spatial reasoning, factual answering and stylistic instruction following — they need different evidence and metrics (see Risks and evaluation). For a survey of the whole design space, see Yin et al. [40] and Caffagni et al. [2].

Adapting a foundation model

Plain version. Use the least expensive adaptation that solves the target task, and check it on data that looks like deployment.

StrategyTrainable componentsUseful starting situation
Prompting / in-context examplesNothingThe model’s interface already fits the task
Frozen features + linear head (linear probe)Small supervised readoutFew labels, good feature transfer
Prompt tuning / visual prompt tuningA few learned input tokensMany tasks share one frozen backbone
Adapters / LoRASmall inserted or low-rank modulesDomain adaptation under memory limits
Partial fine-tuningSelected blocks plus headSome adaptation without full optimizer cost
Full fine-tuning (optionally LP-FT)All model parametersEnough data and compute; transfer alone insufficient

Linear probe

Freeze the encoder ff, compute features once, and fit a linear classifier y^=softmax(Wf(x)+b)\hat y=\mathrm{softmax}(Wf(x)+b). It is cheap, hard to overfit, and is also the standard measurement of representation quality: if a linear readout works, the information is linearly accessible in the features.

Code: why pretrained features help most with few labels

This toy experiment simulates the whole story. Observations are 300-dimensional and mix 8 semantic factors (which determine the class) with 8 large nuisance factors (think lighting and pose) plus noise. “Pretraining” sees 50 upstream classes with plenty of labels and learns an 8-dimensional projection (linear discriminant analysis, standing in for a deep encoder). Downstream, there are four new classes and only a few labels per class.

Code · Python · 35 lines
import numpy as np
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from sklearn.linear_model import LogisticRegression

rng = np.random.default_rng(0)
D, k_sem, k_nui = 300, 8, 8
sem_map = rng.normal(size=(k_sem, D))   # semantic factors -> pixels
nui_map = rng.normal(size=(k_nui, D))   # nuisance factors (light, pose...)

def sample(centers, n):
    y = rng.integers(len(centers), size=n)
    z = centers[y] + 0.7 * rng.normal(size=(n, k_sem))
    u = 4.0 * rng.normal(size=(n, k_nui))          # large nuisance variation
    x = z @ sem_map + u @ nui_map + rng.normal(size=(n, D))
    return x, y

# 1) "Pre-training": many upstream classes with plenty of labels.
up_centers = rng.normal(size=(50, k_sem))
x_up, y_up = sample(up_centers, 10_000)
encoder = LinearDiscriminantAnalysis(n_components=k_sem).fit(x_up, y_up)

# 2) Downstream: four NEW classes, few labels, linear probe only.
new_centers = rng.normal(size=(4, k_sem))
x_te, y_te = sample(new_centers, 2_000)
f_te = encoder.transform(x_te)          # frozen features, computed once
for shots in [1, 5, 20, 100]:
    raw, feat = [], []
    for repeat in range(10):
        x_tr, y_tr = sample(new_centers, 4 * shots)
        raw.append(LogisticRegression(max_iter=3000)
                   .fit(x_tr, y_tr).score(x_te, y_te))
        feat.append(LogisticRegression(max_iter=3000)
                    .fit(encoder.transform(x_tr), y_tr).score(f_te, y_te))
    print(f"{shots:>3} labels/class | raw pixels {np.mean(raw):.3f}"
          f" | frozen features {np.mean(feat):.3f}")

On our machine it prints accuracies of roughly 0.29 vs 0.71 with one label per class, 0.62 vs 0.95 with five, 0.96 vs 0.99 with twenty and 0.98 vs 0.99 with a hundred (raw inputs vs frozen features; chance is 0.25). The pretrained projection already discards the nuisance directions, so a handful of labels suffice; with enough labels, the raw-input probe learns to ignore the nuisance itself and the gap closes. The upstream classes are different from the downstream ones — the features transfer because both share the same semantic factors. If the downstream task depended on a factor that pretraining threw away (here, the nuisance), frozen features would fail no matter how many labels you add. That is the failure mode to test for in practice.

Test accuracy versus labeled examples per class on a log axis; the frozen-feature curve starts near 0.7 at one label and saturates quickly, while the raw-input curve starts near chance and catches up only after about twenty labels.
Figure 5 — The toy transfer experiment over more label budgets (mean ± one standard deviation over 10 draws). Reusable features matter most when labels are scarce.

Fine-tuning, LP-FT and robust fine-tuning

Full fine-tuning updates every weight and usually gives the best in-distribution accuracy when data suffices. But Kumar et al. [32] showed that it can distort good pretrained features: across their benchmarks, fine-tuning gave on average higher in-distribution but lower out-of-distribution accuracy than linear probing. Their explanation is that a randomly initialized head sends large, noisy gradients into the backbone early in training. LP-FT — first fit a linear probe, then fine-tune everything starting from that head — reduced this distortion and outperformed both alternatives in their experiments [32]. WiSE-FT [33] takes another route: linearly interpolate the weights of the zero-shot and fine-tuned models, θγ=(1−γ)θzero-shot+γ θft\theta_\gamma=(1-\gamma)\theta_{\mathrm{zero\text{-}shot}}+\gamma\,\theta_{\mathrm{ft}}, which improved robustness under distribution shift for CLIP-style models while keeping strong target accuracy. Both are cheap to try and worth knowing before declaring that fine-tuning “hurt robustness”.

Parameter-efficient fine-tuning (PEFT)

Adapters [29] insert small bottleneck MLPs inside each Transformer block and train only them, adding a few parameters per task while the original network stays fixed. Prompt tuning learns a few extra input tokens; Visual Prompt Tuning (VPT) [30] does this for ViTs, prepending learnable tokens to the patch sequence of a frozen backbone. Xu et al. [42] review the PEFT landscape with a taxonomy of additive, partial, reparameterized, hybrid and unified fine-tuning, plus experiments on parameter and memory efficiency.

LoRA [10] reparameterizes an update to a frozen weight W0∈Rdout×dinW_0\in\mathbb R^{d_{\mathrm{out}}\times d_{\mathrm{in}}} as

W=W0+αrBA,A∈Rr×din,B∈Rdout×r,W=W_0+\frac{\alpha}{r}BA,\qquad A\in\mathbb R^{r\times d_{\mathrm{in}}},\quad B\in\mathbb R^{d_{\mathrm{out}}\times r},

with rank r≪min⁡(din,dout)r\ll\min(d_{\mathrm{in}},d_{\mathrm{out}}) and a scaling constant α\alpha. Freeze W0W_0 and learn A,BA,B. Trainable weights drop from doutdind_{\mathrm{out}}d_{\mathrm{in}} to r(din+dout)r(d_{\mathrm{in}}+d_{\mathrm{out}}). Initialize B=0B=0 so the initial update is zero, and AA randomly so learning can start. After training, BABA can be merged into W0W_0, so inference costs nothing extra. QLoRA [31] stores the frozen base weights in 4-bit precision and backpropagates through them into LoRA adapters, cutting memory enough to fine-tune very large models on a single GPU.

Code: a low-rank update with a frozen base

This standalone example trains one adapter step on synthetic input. It shows the parameterization and initial behavior; it does not reproduce a full pretrained-model fine-tuning pipeline.

Code · Python · 24 lines
import torch
from torch import nn
from torch.nn import functional as F

torch.manual_seed(0)
class LoRALinear(nn.Module):
    def __init__(self, din, dout, rank=4, alpha=4):
        super().__init__()
        self.base = nn.Linear(din, dout, bias=False)
        self.base.weight.requires_grad_(False)
        self.A = nn.Parameter(torch.randn(rank, din) * .02)
        self.B = nn.Parameter(torch.zeros(dout, rank))
        self.scale = alpha/rank
    def forward(self, x):
        return self.base(x) + self.scale * F.linear(F.linear(x, self.A), self.B)

layer = LoRALinear(32, 16)
x = torch.randn(8, 32)
assert torch.allclose(layer(x), layer.base(x))
opt = torch.optim.Adam([layer.A, layer.B], lr=.01)
loss = F.mse_loss(layer(x), torch.randn(8, 16))
opt.zero_grad(); loss.backward(); opt.step()
assert layer.base.weight.grad is None
print("trainable:", sum(p.numel() for p in layer.parameters() if p.requires_grad))

It prints trainable: 192, which is r(din+dout)=4×(32+16)r(d_{\mathrm{in}}+d_{\mathrm{out}})=4\times(32+16), against 512 for the full matrix. The first assertion checks that the zero-initialized BB leaves the model unchanged at the start; the second, that the frozen base receives no gradient.

Log-scale bars comparing a linear head, low-rank adapters with a head, and full fine-tuning for an illustrative architecture.
Figure 6 — Analytical trainable-weight budgets for an illustrative architecture (feature size 1024, 10 classes, rank 8 on 12 matrices, 100M parameters in total). Activation memory, frozen weight storage and optimizer state must be counted separately.

LoRA is a parameterization, not a guarantee against overfitting or forgetting. Adapters still require domain validation. Frozen BatchNorm statistics, image normalization, output-label mapping and checkpoint-specific preprocessing also need deliberate treatment.

Prompting and in-context learning

The cheapest adaptation changes no weights: write a better text prompt for CLIP, give SAM a box, or put a few solved examples into a multimodal model’s context, as Flamingo did [27]. The vision–language survey by Zhang et al. [41] compares prompting, feature adaptation and fine-tuning across vision tasks. Prompting is fast to iterate but brittle: small wording changes can move results, so fix and report the prompt protocol.

Video, geometry and world models

Plain version. Beyond recognizing a frame, a model may predict change over time, infer three-dimensional structure, or support action planning.

V-JEPA 2 [11] studies self-supervised video models that predict in feature space rather than pixel space, with an action-conditioned extension used for planning. VGGT [12] predicts camera parameters, depth, point maps and tracks from one or many views in a single feed-forward pass. These illustrate two routes beyond classification: temporal state modeling and geometric inference.

A generated video that looks convincing need not obey physical constraints. A scene reconstruction can be useful without generating photorealistic pixels. Measure multi-view consistency, geometry, action-conditioned outcomes and temporal behavior according to the claimed capability. DLCV 13 · World Models develops partial observability, learned dynamics and planning in more detail.

Risks and evaluation

Plain version. Broad pretraining can carry data biases, unsupported predictions and test-set overlap into evaluation. Check these separately from aggregate benchmark performance.

Data provenance, bias and privacy

Web-scale image–text datasets are too large to inspect by hand. LAION-5B [34] made such a dataset public (5.85 billion CLIP-filtered pairs) and thereby enabled open replication such as OpenCLIP [20]; audits of its predecessor LAION-400M by Birhane et al. [35] documented explicit and stereotyping content in image–text pairs. Downstream systems inherit these associations. Licenses and consent also matter: training data, model weights and generated outputs may each carry different terms, and personal data (faces, names, medical images) raises privacy obligations that do not disappear because data was public. SAM’s SA-1B is an example of a dataset whose description emphasizes licensed and privacy-respecting images [5]; record such properties for every model you deploy.

Hallucination

Multimodal LLMs can mention objects that are absent from the image. POPE [37] measures object hallucination with a polling approach: it asks yes/no questions about the presence of specific objects (including frequent and co-occurring distractors), which makes the measurement less dependent on the model’s generation style than scoring free-form captions. A low POPE error does not cover counting, attributes or relations; test those separately.

Contamination and “zero-shot”

If benchmark images (or near-duplicates) are in the pretraining set, a “zero-shot” score partly measures memory. Mayilvahanan et al. [36] asked whether CLIP’s generalization mainly stems from high train–test similarity; after pruning LAION to remove images too similar to common out-of-distribution benchmarks and retraining, performance dropped on some benchmarks but remained strong overall. The lesson is two-sided: similarity explains part of the reported performance, not all of it — so measure it rather than assume either way. Practical checks: deduplicate the test set against available training data, report results on data collected after the model’s data cutoff, and state the full zero-shot protocol (class names, templates, candidate set).

Benchmarks

Aggregate multimodal benchmarks such as MMMU [38] — 11.5K college-level questions from exams, quizzes and textbooks across six disciplines, with charts, diagrams, maps, tables and other image types — test knowledge and reasoning, but a single number mixes perception, OCR, domain knowledge and answer formatting. For a deployment, build a small, task-specific evaluation set with known failure cases, report variance over seeds or prompt wordings, and compare against a simple baseline (a linear probe on frozen features is often surprisingly strong). ML 11 · Evaluation and Deployment covers calibration, splits and deployment in depth.

Modern view

Surveys worth reading.

  • Bommasani et al. [1] — the report that named foundation models. It covers capabilities (language, vision, robotics, reasoning), applications (law, health, education), technology (modeling, training, adaptation, evaluation, data) and societal impact. Its key takeaways — emergence, homogenization, and inherited failure — still frame the field.
  • Awais et al. [39] — a computer-vision survey organizing foundation models by architecture design, training objective (contrastive versus generative), datasets, fine-tuning mechanisms and prompting, and listing open challenges in evaluation, real-world applicability, adversarial robustness and interpretability.
  • Zhang et al. [41] — a TPAMI survey of vision–language models for vision tasks: network architectures, pre-training objectives, downstream tasks, datasets, transfer learning and knowledge distillation, with benchmarking of reviewed methods.
  • Yin et al. [40] and Caffagni et al. [2] — surveys of multimodal large language models: architecture (encoder, connector, LLM), training strategy and data (pre-training, instruction tuning, alignment), evaluation, and hallucination.
  • Han et al. [13] — a survey broadening the map to multimodal generative models, organized by generative category and technique.
  • Xu et al. [42] — a critical review of parameter-efficient fine-tuning that sorts methods into additive, partial, reparameterized, hybrid and unified families and compares representative ones experimentally.

State of the art, 2023–2026. Four trends stand out. (1) Better frozen features: DINOv2 [15] and DINOv3 [16] made self-supervised encoders strong enough that many dense tasks start from frozen patch features. (2) Concept-level prompting: segmentation moved from “click an object” (SAM [5]) to video memory (SAM 2 [6]) to “segment every instance of this phrase” (SAM 3 [7]). (3) Long, interleaved multimodal contexts: models such as Qwen3-VL [9] read many images, video and text together. (4) Beyond recognition: video prediction for planning (V-JEPA 2 [11]) and feed-forward geometry (VGGT [12]) treat foundation models as world and scene estimators.

Open problems.

  • Grounded reliability: reducing hallucination and making models say “not visible” when appropriate [37].
  • Honest evaluation under contamination and benchmark saturation [36].
  • Data: curation as a scaling axis [15], [20], provenance, licensing and consent [34], [35].
  • Efficient adaptation and inference: token budgets for high-resolution images and video; PEFT that preserves robustness [32], [33].
  • Dense, 3D and temporal understanding that matches the semantic strength of image-level models.

For any new “foundation” model, record its input/output interface, training objective, data sources, supported adaptation, checkpoint license, preprocessing and task-level evaluation. This avoids treating model size or an open-ended chat interface as proof of general competence.

Key takeaways

  • Foundation models combine broad pretraining with reuse; the term does not specify one architecture or require text generation.
  • Scaling laws are empirical fits tied to data, objectives and training recipes; extrapolation requires validation.
  • Pretraining objectives shape capabilities but do not uniquely determine downstream performance.
  • Image–text similarity supports retrieval and text-defined classes; similarity is not calibrated confidence.
  • Visual prompts and connectors control information access; report prompt construction and image-resolution constraints.
  • Start with inexpensive adaptation baselines, then test stronger methods for gains, cost, shift sensitivity, hallucination and contamination.

Exercises

  1. Candidate dependence. Add a fourth description to the synthetic CLIP example. Can the softmax “confidence” for an image change without changing the image?
Hint

Yes: the denominator of the softmax changes. Similarity to a fixed description is unchanged, but the normalized probability depends on the candidate set. Try adding a text vector close to the ambiguous image’s embedding.

  1. Compute-optimal allocation. Using the parametric law L(N,D)=E+A/Nα+B/DβL(N,D)=E+A/N^{\alpha}+B/D^{\beta} and C=6NDC=6ND, show that the optimal model size grows as N⋆∝Cβ/(α+β)N^\star\propto C^{\beta/(\alpha+\beta)}. What does α=β\alpha=\beta imply?
Hint

Substitute D=C/(6N)D=C/(6N), differentiate with respect to NN and set to zero: αAN−α−1=βB(6N/C)β/N\alpha A N^{-\alpha-1}=\beta B (6N/C)^{\beta}/N. Solving gives Nα+β∝CβN^{\alpha+\beta}\propto C^{\beta}. With α=β\alpha=\beta, N⋆∝C1/2N^\star\propto C^{1/2} and therefore D⋆∝C1/2D^\star\propto C^{1/2} too: scale parameters and tokens equally, as Hoffmann et al. concluded.

  1. Break the transfer. In the linear-probe experiment, make the downstream class depend on the nuisance factors instead of the semantic ones. What happens to the frozen-feature probe as labels increase, and what does that say about choosing a foundation model?
Hint

Draw class centers for u instead of z. The frozen 8-D features were trained to discard the nuisance directions, so they stay near chance however many labels you add, while the raw-input probe improves. A backbone is only useful if its pretraining kept the information your task needs — check with a linear probe before investing in adaptation.

  1. LoRA budget. A ViT-B block has four 768×768768\times768 attention projections and two MLP matrices of sizes 3072×7683072\times768 and 768×3072768\times3072. Count trainable weights for full fine-tuning of these six matrices and for rank-8 LoRA on all of them.
Hint

Full: 4⋅7682+2⋅3072⋅768=2,359,296+4,718,592=7,077,8884\cdot768^2+2\cdot3072\cdot768=2{,}359{,}296+4{,}718{,}592=7{,}077{,}888. LoRA: 8⋅[4⋅(768+768)+2⋅(3072+768)]=8⋅(6144+7680)=110,5928\cdot[4\cdot(768+768)+2\cdot(3072+768)]=8\cdot(6144+7680)=110{,}592, about 1.6% of the full count.

  1. Prompt fairness. Design a SAM evaluation that separates mask quality from prompt quality.
Hint

Fix the prompt protocol and count human interactions. Evaluate automatic proposals separately from masks produced with ideal ground-truth prompts, and report both. For SAM 3, compare noun-phrase prompts with exemplar prompts on the same images.

  1. Contamination audit. You are given a new image–text model that reports high zero-shot accuracy on a public benchmark. List three checks you would run before trusting the number.
Hint

(a) Near-duplicate search of test images against any available training data (e.g. by embedding similarity); (b) evaluation on a freshly collected set from after the data cutoff; (c) sensitivity to the prompt templates and the candidate class list; optionally compare with a linear probe and a simple baseline.

Paper guides

1 paper guide, each on a page of its own.

Open the paper guides →

References

  1. R. Bommasani et al., “On the Opportunities and Risks of Foundation Models,” Stanford CRFM report, arXiv:2108.07258, 2021; revised 2022. arXiv
  2. D. Caffagni et al., “The Revolution of Multimodal Large Language Models: A Survey,” Findings of ACL, 2024. arXiv
  3. A. Radford et al., “Learning Transferable Visual Models From Natural Language Supervision,” ICML, 2021. arXiv
  4. X. Zhai, B. Mustafa, A. Kolesnikov and L. Beyer, “Sigmoid Loss for Language Image Pre-Training,” ICCV, 2023. arXiv
  5. A. Kirillov et al., “Segment Anything,” ICCV, 2023. arXiv
  6. N. Ravi et al., “SAM 2: Segment Anything in Images and Videos,” ICLR, 2025. arXiv
  7. N. Carion et al., “SAM 3: Segment Anything with Concepts,” arXiv:2511.16719, 2025; revised March 2026. arXiv
  8. H. Liu, C. Li, Q. Wu and Y. J. Lee, “Visual Instruction Tuning,” NeurIPS, 2023. arXiv
  9. S. Bai et al., “Qwen3-VL Technical Report,” arXiv:2511.21631, 2025. arXiv
  10. E. J. Hu et al., “LoRA: Low-Rank Adaptation of Large Language Models,” ICLR, 2022. arXiv
  11. M. Assran et al., “V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning,” arXiv:2506.09985, 2025. arXiv
  12. J. Wang et al., “VGGT: Visual Geometry Grounded Transformer,” CVPR, 2025. arXiv
  13. L. Han et al., “A Survey of Generative Categories and Techniques in Multimodal Generative Models,” arXiv:2506.10016, 2025. arXiv
  14. C. M. Bishop and H. Bishop, Deep Learning: Foundations and Concepts, Springer, 2024. Book
  15. M. Oquab et al., “DINOv2: Learning Robust Visual Features without Supervision,” Transactions on Machine Learning Research, 2024. arXiv
  16. O. Siméoni et al., “DINOv3,” arXiv:2508.10104, 2025. arXiv
  17. J. Kaplan et al., “Scaling Laws for Neural Language Models,” arXiv:2001.08361, 2020. arXiv
  18. J. Hoffmann et al., “Training Compute-Optimal Large Language Models,” arXiv:2203.15556, 2022; published as “An empirical analysis of compute-optimal large language model training,” NeurIPS, 2022. arXiv
  19. X. Zhai, A. Kolesnikov, N. Houlsby and L. Beyer, “Scaling Vision Transformers,” CVPR, 2022. arXiv
  20. M. Cherti et al., “Reproducible Scaling Laws for Contrastive Language-Image Learning,” CVPR, 2023. arXiv
  21. A. Dosovitskiy et al., “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale,” ICLR, 2021. arXiv
  22. K. He, X. Chen, S. Xie, Y. Li, P. Dollár and R. Girshick, “Masked Autoencoders Are Scalable Vision Learners,” CVPR, 2022. arXiv
  23. H. Bao, L. Dong, S. Piao and F. Wei, “BEiT: BERT Pre-Training of Image Transformers,” ICLR, 2022. arXiv
  24. M. Caron et al., “Emerging Properties in Self-Supervised Vision Transformers,” ICCV, 2021. arXiv
  25. C. Jia et al., “Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision,” ICML, 2021. arXiv
  26. J. Li, D. Li, S. Savarese and S. Hoi, “BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models,” ICML, 2023. PMLR
  27. J.-B. Alayrac et al., “Flamingo: a Visual Language Model for Few-Shot Learning,” NeurIPS, 2022. arXiv
  28. H. Liu, C. Li, Y. Li and Y. J. Lee, “Improved Baselines with Visual Instruction Tuning,” CVPR, 2024. arXiv
  29. N. Houlsby et al., “Parameter-Efficient Transfer Learning for NLP,” ICML, 2019. PMLR
  30. M. Jia et al., “Visual Prompt Tuning,” ECCV, 2022. arXiv
  31. T. Dettmers, A. Pagnoni, A. Holtzman and L. Zettlemoyer, “QLoRA: Efficient Finetuning of Quantized LLMs,” NeurIPS, 2023. arXiv
  32. A. Kumar, A. Raghunathan, R. Jones, T. Ma and P. Liang, “Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution,” ICLR, 2022. arXiv
  33. M. Wortsman et al., “Robust Fine-Tuning of Zero-Shot Models,” CVPR, 2022. arXiv
  34. C. Schuhmann et al., “LAION-5B: An Open Large-Scale Dataset for Training Next Generation Image-Text Models,” NeurIPS Datasets and Benchmarks Track, 2022. arXiv
  35. A. Birhane, V. U. Prabhu and E. Kahembwe, “Multimodal Datasets: Misogyny, Pornography, and Malignant Stereotypes,” arXiv:2110.01963, 2021. arXiv
  36. P. Mayilvahanan, T. Wiedemer, E. Rusak, M. Bethge and W. Brendel, “Does CLIP’s Generalization Performance Mainly Stem from High Train-Test Similarity?,” ICLR, 2024. arXiv
  37. Y. Li et al., “Evaluating Object Hallucination in Large Vision-Language Models,” EMNLP, 2023. arXiv
  38. X. Yue et al., “MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI,” CVPR, 2024. arXiv
  39. M. Awais et al., “Foundational Models Defining a New Era in Vision: A Survey and Outlook,” arXiv:2307.13721, 2023. arXiv
  40. S. Yin et al., “A Survey on Multimodal Large Language Models,” National Science Review, 2024. arXiv
  41. J. Zhang, J. Huang, S. Jin and S. Lu, “Vision-Language Models for Vision Tasks: A Survey,” IEEE TPAMI, vol. 46, no. 8, 2024. arXiv
  42. L. Xu et al., “Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment,” arXiv:2312.12148, 2023. arXiv
  43. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li and L. Fei-Fei, “ImageNet: A Large-Scale Hierarchical Image Database,” CVPR, 2009, pp. 248–255. doi
  44. J. Devlin, M.-W. Chang, K. Lee and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” NAACL-HLT, 2019, pp. 4171–4186. doi
  45. OpenAI, “GPT-4 Technical Report,” arXiv:2303.08774, 2023. arXiv

Check your understanding

This tutorial has 6 exercises. Can you solve them without opening the hints?

Go to the exercises ↑

Next ML 11 · Evaluation, Efficiency and Deployment · 30 min