Full search page →

ML 06 · Representation Learning and Self-Supervision

Machine LearningIntermediate27 minUpdated Nov 6, 2026

Needs: ML 05 · Attention and Vision Transformers

Navigate: this page and this track: Navigate: this page and this track
Explore this chapter 1 interactive experiment

Move paired and unpaired unit vectors, then inspect the normalized matching loss used by contrastive learning.

What you’ll learn

  • Define useful representations and task-dependent invariances.
  • Reuse features through nearest neighbors, linear probes or fine-tuning.
  • Compare contrastive, teacher–student, masked and feature-prediction objectives.
  • Explain representation collapse and mechanisms that discourage it.
  • Evaluate transfer and image–text alignment under explicit protocols.

The big picture

Suppose you have a million unlabeled images but only 500 labeled examples for the task you care about. Training a large network from scratch on those 500 examples ignores almost all the available data. Representation learning splits the job in two. First, learn an encoder h=fθ(x)h=f_\theta(x) from a broad signal that does not need your labels. Second, train a small task-specific head on top of hh using the few labels you have.

Why it matters: almost every modern vision system starts from a pretrained encoder. Image classifiers, detectors, segmentation models, image–text models and the image encoders inside generative models are built on representations learned this way. The quality of that starting point often matters more than the details of the task head.

This chapter starts with what a representation is and why transfer works, then walks through the self-supervised objectives that dominate practice, and ends with how to measure whether a representation is any good. It assumes you know neural networks (ML 02), CNNs (ML 03) and vision transformers (ML 05). Linear factor models and autoencoders, the classical route to representations, have their own chapter: ML 06A.

What is a representation?

Plain version. A representation is the list of numbers a model uses to describe an input. A good one makes the things you care about easy to read off, and hides the things you do not care about.

Precise version. An encoder is a function fθ:X→RDf_\theta:\mathcal X\to\mathbb R^D from inputs xx (here, images) to feature vectors h=fθ(x)h=f_\theta(x), where θ\theta are the learnable parameters and DD is the feature dimension. A task head gg maps hh to a prediction, so the full model is g∘fθg\circ f_\theta. We call hh a good representation for a task when a simple head, such as a linear classifier, can solve the task from hh using few labeled examples. Bengio, Courville and Vincent’s review [1] is the classic statement of this view; it lists the “priors” that make representations useful, such as smoothness, multiple explanatory factors, a hierarchy of increasingly abstract features, and shared factors across tasks.

Local versus distributed representations

The simplest representation of a category is one-hot: one unit per concept, with exactly one unit on. It is easy to read but it cannot generalize. Knowing about “red car” says nothing about “red truck”, because they share no active units. A distributed representation describes each input with many features that are each shared by many inputs: “red”, “blue”, “car”, “truck”. Now “red truck” reuses the “red” unit learned from red cars.

Two 4 by 4 grids. Left: one-hot codes with a single active unit per concept. Right: distributed codes where red car, blue car, red truck and blue truck each activate one color unit and one vehicle unit.
Figure 1 — Local (one-hot) versus distributed codes for four concepts. With distributed codes, a new combination such as a red truck reuses features learned from other examples.

The counting argument is striking. With DD binary features, a one-hot code distinguishes DD concepts, while a distributed code can in principle distinguish 2D2^D configurations [1]. Deep networks learn distributed codes automatically: each hidden unit responds to many inputs, and each input activates many units. Most of the practical value of representation learning comes from this sharing.

Invariance and equivariance

A useful representation keeps differences relevant to the task and suppresses differences that distract from it.

  • A feature is invariant to a transformation TT if f(T(x))=f(x)f(T(x))=f(x): shifting a cat by two pixels should not change “this is a cat”.
  • A feature is equivariant if it changes in a predictable way, f(T(x))=T′(f(x))f(T(x))=T'(f(x)): shifting the cat should shift its segmentation mask by the same amount.

“Useful” always depends on the downstream task. Color invariance helps object recognition but hurts fruit-ripeness estimation. Rotation invariance helps for textures but confuses 6 and 9. Dense prediction (segmentation, depth) needs spatially equivariant patch features, not a single invariant vector per image. We will see that self-supervised objectives choose their invariances mostly through the data augmentations they use, so this choice is a modeling decision, not a detail.

Learning distances: metric learning

PCA, introduced in ML 01 and developed in ML 06A, keeps the directions of highest variance. But high variance may come from lighting rather than object identity. Supervised learning shapes features around labels; metric learning shapes distances around pairs that should match. For an anchor aa, a positive pp (same identity) and a negative nn (different identity), the triplet loss is

Ltriplet=max⁡(0, ∥ha−hp∥2−∥ha−hn∥2+m),\mathcal{L}_{\mathrm{triplet}}= \max\big(0,\ \|h_a-h_p\|^2-\|h_a-h_n\|^2+m\big),

where ha,hp,hnh_a, h_p, h_n are the three feature vectors and m>0m>0 is a margin. The loss is zero once the positive is closer than the negative by at least mm. An easy negative gives zero loss and teaches nothing, so choosing informative (“hard”) negatives matters. Contrastive self-supervised learning, below, is metric learning in which the positives are created by augmentation instead of by labels.

Transfer learning: pretrain once, reuse many times

Plain version. Train a network on a big task, keep its feature extractor, and reuse it for a small task. You can freeze the features and only train a new head, or let the whole network keep learning.

Precise version. Pretraining finds encoder parameters θ⋆\theta^\star on a source dataset or objective. Transfer to a target task with labeled set {(xi,yi)}i=1n\{(x_i,y_i)\}_{i=1}^{n} uses one of three protocols:

  1. kk-nearest neighbors (kk-NN): no training. Classify a test feature fθ⋆(x)f_{\theta^\star}(x) by the labels of its kk most similar training features.
  2. Linear probe: keep θ⋆\theta^\star frozen and fit a linear classifier y^=softmax⁡(Wh+b)\hat y=\operatorname{softmax}(Wh+b) on h=fθ⋆(x)h=f_{\theta^\star}(x), where WW and bb are the only trained parameters.
  3. Fine-tuning: start from θ⋆\theta^\star and update the encoder together with the head, usually with a small learning rate.
Diagram. Stage 1: large unlabeled data feeds a pretext objective that produces an encoder f. Stage 2: small labeled data and the encoder feed either a k-NN or linear probe with f frozen, or fine-tuning with f updated, producing task predictions.
Figure 2 — The two-stage pattern of representation learning. Pretraining is expensive and done once; transfer is cheap and done per task.

Why does transfer work at all? Yosinski et al. [2] measured it layer by layer on ImageNet-trained CNNs. First-layer features (edge and color detectors much like Gabor filters) were general: they transferred well between tasks. Features became progressively more specific to the source task in higher layers, and transferability decreased as the source and target tasks became more different. They also found that initializing from transferred features and fine-tuning improved generalization even on the target task, compared with training from random initialization.

For years, the default pretraining source was supervised ImageNet classification. Labels are expensive, though, and a label like “dog” tells the network nothing about the dog’s pose, the grass or the lighting. Self-supervised learning removes the label bottleneck by making the data supervise itself.

Self-supervised learning: let the data make the labels

Plain version. Build prediction targets from the data, such as matching two views or recovering masked content. Whether the learned features transfer still needs testing.

Precise version. A self-supervised objective is a loss L(θ;x)\mathcal L(\theta;x) computed from unlabeled inputs alone (or from naturally paired data such as image–caption pairs). The surveys by Liu et al. [3] and Gui et al. [4] and the practical “cookbook” by Balestriero et al. [5] organize the field into the families summarized here:

FamilyTraining signalRepresentative methodsMain risk
ContrastivePull positive pairs together, push other samples apartSimCLR, MoCoNeeds many negatives; false negatives
Self-distillation (no negatives)Student matches a teacher on another viewBYOL, SimSiam, DINOCollapse without asymmetry or centering
Redundancy reductionAgreement plus decorrelated, non-constant feature dimensionsBarlow Twins, VICRegExtra hyperparameters for the regularizers
Masked modelingPredict hidden pixels or tokensBEiT, MAEFeatures may focus on low-level detail
Feature predictionPredict features of hidden regionsI-JEPA, V-JEPA 2Needs anti-collapse design like distillation
Language supervisionMatch images with their captionsCLIP, SigLIPInherits web-data biases and noise

The families are learning objectives, not architectures: each can train a CNN or a transformer. Modern systems often combine several of them. The sections below take them in turn.

Contrastive learning and the InfoNCE loss

Plain version. Make two augmented versions of the same image agree, and make them disagree with every other image in the batch. It is a multiple-choice question: “out of these NN candidates, which one is my partner?”

InfoNCE

Van den Oord, Li and Vinyals introduced the InfoNCE loss in Contrastive Predictive Coding (CPC) [6]. Take a query qq, one positive key k+k^+ that “belongs” to it, and N−1N-1 negative keys k1−,…,kN−1−k^-_1,\ldots,k^-_{N-1}. With a similarity score s(⋅,⋅)s(\cdot,\cdot) and a temperature τ>0\tau>0,

LInfoNCE=−log⁡exp⁡(s(q,k+)/τ)exp⁡(s(q,k+)/τ)+∑j=1N−1exp⁡(s(q,kj−)/τ).\mathcal L_{\mathrm{InfoNCE}}= -\log\frac{\exp\big(s(q,k^+)/\tau\big)} {\exp\big(s(q,k^+)/\tau\big)+\sum_{j=1}^{N-1}\exp\big(s(q,k^-_j)/\tau\big)}.

This is exactly softmax cross-entropy for an NN-way classification problem whose correct class is the positive. Two consequences follow.

  • A lower bound on mutual information. CPC shows that I(q;k+)≥log⁡N−LInfoNCEI(q;k^+)\ge\log N-\mathcal L_{\mathrm{InfoNCE}}, where II is mutual information. Minimizing the loss raises this lower bound, and more negatives (larger NN) allow a higher bound. In practice this bound is loose, and the quality of the features depends heavily on the encoder and augmentations, so read it as motivation rather than as the explanation of why the features are good.
  • The temperature controls hardness. Dividing by a small τ\tau sharpens the softmax, so the loss concentrates on the negatives most similar to the query (the hard negatives).

SimCLR

SimCLR [7] is InfoNCE in its simplest form. Take a batch of BB images and produce two augmented views of each (random crop and resize, color distortion, blur), giving 2B2B views. A backbone ff maps each view to hh, a small MLP projection head gg maps hh to z=g(h)z=g(h), and the similarity is cosine similarity s(zi,zj)=zi⊤zj/(∥zi∥∥zj∥)s(z_i,z_j)=z_i^\top z_j/(\|z_i\|\|z_j\|). For view ii with partner jj,

ℓi,j=−log⁡exp⁡(zi⊤zj/τ)∑k≠iexp⁡(zi⊤zk/τ),\ell_{i,j}=-\log \frac{\exp(z_i^\top z_j/\tau)} {\sum_{k\ne i}\exp(z_i^\top z_k/\tau)},

where the zz are ℓ2\ell_2-normalized and the sum runs over all 2B−12B-1 other views. Every other image in the batch is a negative, so no memory bank is needed, but the method benefits from large batches. The total loss averages ℓi,j\ell_{i,j} over both orderings of every positive pair; the paper calls this the NT-Xent loss (normalized temperature-scaled cross-entropy).

Three findings from the SimCLR paper are worth remembering [7]. The composition of augmentations matters a lot, with random cropping plus color distortion being especially important. The projection head helps: features before the head (hh) transfer better than those after it (zz), so the head is discarded after pretraining. And contrastive learning benefits from larger batches and longer training than supervised learning.

Code: check the loss and its collapse value

This standalone PyTorch example uses synthetic paired vectors to verify the loss, the self-mask and the gradients. It does not train an encoder.

Code · Python · 23 lines
import torch
from torch.nn import functional as F

torch.manual_seed(0)
B, D = 8, 16
base = torch.randn(B, D)
views = torch.cat([base + .1*torch.randn_like(base),
                   base + .1*torch.randn_like(base)]).requires_grad_()
z = F.normalize(views, dim=-1)
logits = z @ z.T / .2
self_mask = torch.eye(2*B, dtype=torch.bool)
logits = logits.masked_fill(self_mask, float("-inf"))
positive = (torch.arange(2*B) + B) % (2*B)
loss = F.cross_entropy(logits, positive)
loss.backward()
assert torch.isfinite(views.grad).all()
print("paired loss:", round(loss.item(), 4))

collapsed = F.normalize(torch.ones(2*B, D), dim=-1)
scores = (collapsed @ collapsed.T / .2).masked_fill(self_mask, float("-inf"))
collapsed_loss = F.cross_entropy(scores, positive)
print("collapsed loss:", round(collapsed_loss.item(), 4))
print("expected:", round(torch.log(torch.tensor(2*B-1.)).item(), 4))

It prints a paired loss of 0.1423 and a collapsed loss of 2.7081, equal to log⁡15\log 15. If every representation is identical, the positive gets probability 1/(2B−1)1/(2B-1) and the loss is log⁡(2B−1)\log(2B-1), the worst value a uniform guess can get. Contrastive discrimination therefore makes complete collapse a bad solution, although optimization and negative selection can still be difficult.

A 16 by 16 heatmap of cosine similarities. Bright yellow appears on the main diagonal and on the two off-diagonals that pair view a with view b of the same image; most other entries are near zero.
Figure 3 — Cosine similarities between 16 views (8 digit images, two augmentations each) after training the small SimCLR encoder used later in this chapter. The bright off-diagonal stripes are the positive pairs; InfoNCE pushes them up and everything else down.

MoCo: a queue of negatives and a momentum encoder

InfoNCE prefers many negatives, but SimCLR gets them only from the current batch. Momentum Contrast (MoCo) [8] decouples the number of negatives from the batch size with two ideas.

  1. A queue. Keys encoded in recent batches are stored in a first-in, first-out queue of length KK and reused as negatives. The query sees KK negatives no matter how small the batch is.
  2. A momentum encoder. Keys in the queue were produced by slightly older networks. If the key encoder changed quickly, old and new keys would be inconsistent. MoCo therefore computes keys with a separate encoder whose parameters θk\theta_k follow the query encoder’s parameters θq\theta_q slowly:
θk←m θk+(1−m) θq,\theta_k\leftarrow m\,\theta_k+(1-m)\,\theta_q,

where m∈[0,1)m\in[0,1) is the momentum, close to 1 (the paper’s default is 0.999). Only θq\theta_q receives gradients. This exponential moving average (EMA) teacher idea reappears in BYOL, DINO and JEPA.

Code · Python · 21 lines
import torch
from torch.nn import functional as F

torch.manual_seed(0)
B, D, K = 8, 16, 32          # batch, embedding size, queue length

# --- MoCo-style InfoNCE: one positive key per query, negatives from a queue
q = F.normalize(torch.randn(B, D), dim=1)                 # queries (online encoder)
k = F.normalize(q + 0.1 * torch.randn(B, D), dim=1)       # keys (momentum encoder)
queue = F.normalize(torch.randn(K, D), dim=1)             # keys from past batches
l_pos = (q * k).sum(1, keepdim=True)                      # B x 1
l_neg = q @ queue.T                                       # B x K
logits = torch.cat([l_pos, l_neg], dim=1) / 0.07
loss_moco = F.cross_entropy(logits, torch.zeros(B, dtype=torch.long))
queue = torch.cat([k, queue])[:K]                         # enqueue new, drop oldest
print("MoCo logits:", tuple(logits.shape), "loss:", round(loss_moco.item(), 3))

# momentum (EMA) update of the key encoder's parameters
theta_q, theta_k, m = torch.randn(5), torch.zeros(5), 0.999
theta_k = m * theta_k + (1 - m) * theta_q
print("key params moved by:", round((theta_k.norm() / theta_q.norm()).item(), 4))

Each query gets a 1+K1+K-way classification problem whose correct answer is always column 0, hence the all-zero targets. The output shows logits of shape (8, 33), and the EMA step moves the key encoder only 0.1% of the way toward the query encoder (from zero, after one step). In a real implementation, qq and kk come from the two encoders applied to two augmentations of the same images.

Alignment and uniformity

Wang and Isola [9] gave a clean geometric reading of what contrastive losses do on the unit sphere. With normalized features, the loss asymptotically optimizes two properties:

Lalign=E(x,x+)∥f(x)−f(x+)∥22,Lunif=log⁡Ex,y e−t∥f(x)−f(y)∥22,\mathcal L_{\mathrm{align}}=\mathbb E_{(x,x^+)}\big\|f(x)-f(x^+)\big\|_2^2, \qquad \mathcal L_{\mathrm{unif}}=\log\mathbb E_{x,y}\,e^{-t\|f(x)-f(y)\|_2^2},

where (x,x+)(x,x^+) is a positive pair, x,yx,y are independent samples, and t>0t>0 (commonly t=2t=2). Alignment is low when positive pairs map to nearby points. Uniformity is low when features spread out over the sphere, so they keep as much information about the input as possible. A collapsed encoder has perfect alignment (0) and the worst uniformity (0); a useful encoder has both small.

Code · Python · 18 lines
import torch
from torch.nn import functional as F

def alignment(z1, z2):            # mean squared distance between positive pairs
    return (z1 - z2).square().sum(1).mean()

def uniformity(z, t=2.0):         # log of mean Gaussian potential over all pairs
    return torch.pdist(z).square().mul(-t).exp().mean().log()

torch.manual_seed(0)
n, D = 512, 16
collapsed = F.normalize(torch.ones(n, D), dim=1)
spread = F.normalize(torch.randn(n, D), dim=1)
noisy_view = F.normalize(spread + 0.1 * torch.randn(n, D), dim=1)
print("collapsed: align", alignment(collapsed, collapsed).item(),
      "uniform", round(uniformity(collapsed).item(), 3))
print("spread:    align", round(alignment(spread, noisy_view).item(), 3),
      "uniform", round(uniformity(spread).item(), 3))

The collapsed set scores alignment 0.0 and uniformity 0.0; the spread set scores about 0.138 and −3.51. Figure 4 tracks the same two quantities while training the digit encoder below. At step 0 the random network maps every view to almost the same point, so alignment is near zero only because everything has collapsed. Training first spreads the features out (uniformity drops sharply), then gradually pulls positives back together.

Two line plots over 800 training steps. Left: alignment starts near zero, jumps to about 0.7, then trends down noisily to about 0.2. Right: uniformity falls from about 0 to about minus 3.3.
Figure 4 — Alignment and uniformity of the projection features during SimCLR training on 8×8 digits. Low alignment alone is meaningless (a constant encoder has it); the useful regime is low alignment together with low uniformity.

A worked example: SimCLR on 8×8 digits

Plain version. Train a tiny SimCLR encoder on scikit-learn’s digits without using any labels, then check whether its features make a nearest-neighbor classifier more robust.

The augmentation here is a random shift of up to one pixel, a random brightness change and pixel noise. We therefore expect the learned features to be roughly invariant to small shifts, something raw pixels are not. The evaluation uses the same labeled training set for every feature type and tests on clean digits and on digits shifted by up to one pixel.

Code · Python · 55 lines
import copy
import numpy as np
import torch
from torch import nn
from torch.nn import functional as F
from sklearn.datasets import load_digits
from sklearn.decomposition import PCA
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier

torch.manual_seed(0); np.random.seed(0); torch.set_num_threads(1)
d = load_digits()
Xtr, Xte, ytr, yte = train_test_split(d.data / 16., d.target, test_size=.3,
                                      random_state=0, stratify=d.target)
Xtr_t = torch.tensor(Xtr, dtype=torch.float32)

def augment(x):                      # the augmentations define the invariances
    img = F.pad(x.view(-1, 1, 8, 8), (1, 1, 1, 1))
    i, j = np.random.randint(0, 3, size=2)               # shift by up to 1 px
    img = img[:, :, i:i + 8, j:j + 8]
    img = img * (0.7 + 0.6 * torch.rand(len(img), 1, 1, 1))  # brightness
    return (img + 0.1 * torch.randn_like(img)).reshape(len(x), 64)  # noise

backbone = nn.Sequential(nn.Unflatten(1, (1, 8, 8)),
                         nn.Conv2d(1, 32, 3, padding=1), nn.ReLU(),
                         nn.Conv2d(32, 64, 3, padding=1), nn.ReLU(),
                         nn.AdaptiveAvgPool2d(2), nn.Flatten())  # h: 256-d
random_backbone = copy.deepcopy(backbone)                    # baseline
head = nn.Sequential(nn.ReLU(), nn.Linear(256, 64))          # projection g
opt = torch.optim.Adam([*backbone.parameters(), *head.parameters()], lr=1e-3)

def nt_xent(z1, z2, tau=0.2):
    z = F.normalize(torch.cat([z1, z2]), dim=-1)
    n = len(z1)
    logits = (z @ z.T / tau).fill_diagonal_(float("-inf"))
    return F.cross_entropy(logits, (torch.arange(2 * n) + n) % (2 * n))

for step in range(800):                                 # about a minute on CPU
    x = Xtr_t[torch.randint(len(Xtr_t), (256,))]
    loss = nt_xent(head(backbone(augment(x))), head(backbone(augment(x))))
    opt.zero_grad(); loss.backward(); opt.step()

def shift(x, dy, dx):                   # move one 8x8 image, fill with zeros
    img = np.zeros((10, 10)); img[1:9, 1:9] = x.reshape(8, 8)
    return img[1 - dy:9 - dy, 1 - dx:9 - dx].ravel()

moves = np.random.default_rng(0).integers(-1, 2, size=(len(Xte), 2))
Xte_shift = np.stack([shift(x, *m) for x, m in zip(Xte, moves)])
enc = lambda net: (lambda A: net(torch.tensor(A, dtype=torch.float32)).detach().numpy())
pca = PCA(32, random_state=0).fit(Xtr)
for name, f in [("raw pixels", lambda A: A), ("PCA-32", pca.transform),
                ("random CNN", enc(random_backbone)), ("SimCLR CNN", enc(backbone))]:
    knn = KNeighborsClassifier(5, metric="cosine").fit(f(Xtr), ytr)
    print(f"{name:11s} clean {knn.score(f(Xte), yte):.3f}  "
          f"shifted {knn.score(f(Xte_shift), yte):.3f}")

On our machine this prints:

Code · 4 lines
raw pixels  clean 0.978  shifted 0.524
PCA-32      clean 0.978  shifted 0.507
random CNN  clean 0.974  shifted 0.524
SimCLR CNN  clean 0.987  shifted 0.919

Exact numbers vary slightly across PyTorch versions and hardware. The pattern is the lesson. On clean digits, raw pixels are already excellent: this dataset is easy, and a fancy encoder adds little. On shifted digits, raw pixels, PCA and an untrained CNN of the same architecture all drop to about 50%, while the self-supervised features keep most of their accuracy. The network learned the invariance we asked for through the augmentations, and nothing more: had we tested on rotated digits, which the augmentations never showed, we should not expect the same robustness.

Bar chart of 5-NN accuracy for raw pixels, PCA, random CNN and SimCLR CNN. All are about 0.97 to 0.98 on clean digits. On shifted digits the first three fall to about 0.51 to 0.52, while SimCLR CNN stays at 0.93.
Figure 5 — Same labels, same classifier, different features. The figure script uses a separate random stream, so its numbers differ from the printed output by up to 0.01. The random CNN rules out the explanation that convolution alone produces the robustness.

Optional: SimCLR (in the Paper guides). Read the original figure, evidence, design choices and limitations.

Without negatives: why matching alone can collapse

Plain version. If the only instruction is “make the two outputs agree”, returning the same constant for every image solves it perfectly. Methods without negatives need some other mechanism to stop this.

Minimizing ∥fθ(v1)−fθ(v2)∥2\|f_\theta(v_1)-f_\theta(v_2)\|^2 over two views v1,v2v_1,v_2 has the trivial solution fθ(x)=cf_\theta(x)=c for a constant cc. Agreement alone does not require the features to carry any information. Complete collapse maps everything to one point; dimensional collapse is subtler: the features vary, but only along a few directions, wasting most of the DD dimensions.

Three synthetic embedding diagrams: all samples at one point, paired views crowded together, and paired views spread around a circle.
Figure 6 — Schematic of collapse, alignment without spread, and alignment with spread. The points are constructed illustrations, not embeddings learned by a published model.

BYOL: an online network chases a slow target

BYOL [10] uses two networks. The online network (encoder, projector and an extra predictor qϕq_\phi) is trained to predict the target network’s projection of a different view of the same image. The target is not trained by gradients; it is an EMA of the online weights, as in MoCo:

θtarget←m θtarget+(1−m) θonline.\theta_{\mathrm{target}}\leftarrow m\,\theta_{\mathrm{target}}+(1-m)\,\theta_{\mathrm{online}}.

The loss is the squared distance between the normalized prediction and the normalized target projection, symmetrized over the two views. The target output is treated as a constant (stop-gradient). There are no negatives, yet BYOL does not collapse in practice. The asymmetry (predictor on one side, stop-gradient and EMA on the other) and the full training recipe matter; EMA alone does not prove collapse impossible.

SimSiam: stop-gradient is the key ingredient

Chen and He’s SimSiam [11] stripped BYOL down further: one shared encoder for both views, a predictor hh on one branch, no momentum encoder, no negatives and no large batches. With p1=h(f(v1))p_1=h(f(v_1)) and z2=f(v2)z_2=f(v_2),

L=12 D(p1,sg⁡(z2))+12 D(p2,sg⁡(z1)),D(p,z)=−p⊤z∥p∥ ∥z∥,\mathcal L=\tfrac12\,\mathcal D\big(p_1,\operatorname{sg}(z_2)\big)+\tfrac12\,\mathcal D\big(p_2,\operatorname{sg}(z_1)\big), \qquad \mathcal D(p,z)=-\frac{p^\top z}{\|p\|\,\|z\|},

where sg⁡\operatorname{sg} is stop-gradient and D\mathcal D is negative cosine similarity. The paper’s experiments show that removing the stop-gradient makes the model collapse quickly, while the predictor is also needed. The authors interpret the method as an alternating, expectation-maximization-like optimization. The practical lesson: very small architectural asymmetries decide whether these methods learn or collapse.

DINO: self-distillation with centering and sharpening

DINO [12] turns the agreement problem into classification with soft labels. Teacher and student both output a probability vector over KK “prototype” dimensions via a temperature-scaled softmax. The student, seeing a view x′x', is trained to match the teacher’s distribution on another view xx:

L=−∑k=1KPt(x)k log⁡Ps(x′)k,Pt(x)=softmax⁡ ⁣(gt(x)−cτt),\mathcal L=-\sum_{k=1}^{K}P_t(x)_k\,\log P_s(x')_k, \qquad P_t(x)=\operatorname{softmax}\!\Big(\frac{g_t(x)-c}{\tau_t}\Big),

where gtg_t is the teacher network (an EMA of the student), cc is a running mean of teacher outputs (centering) and τt\tau_t is a low teacher temperature (sharpening). Centering stops one dimension from dominating; sharpening stops the output from becoming uniform. Each alone collapses in a different way; together they balance. DINO also uses multi-crop: the teacher sees only large global crops, while the student also sees small local crops, so the student must infer global content from local evidence. With vision transformers, DINO’s attention maps were notable for highlighting object boundaries without any segmentation labels.

Barlow Twins and VICReg: regularize the statistics directly

A different route is to forbid collapse explicitly in the loss. Barlow Twins [13] computes the D×DD\times D cross-correlation matrix CC between the (batch-standardized) embeddings of the two views and pushes it toward the identity:

LBT=∑i(1−Cii)2+λ∑i∑j≠iCij2,\mathcal L_{\mathrm{BT}}=\sum_i(1-C_{ii})^2+\lambda\sum_i\sum_{j\ne i}C_{ij}^2,

where the first term makes each dimension agree across views and the second, weighted by λ>0\lambda>0, decorrelates different dimensions so they do not carry redundant information. VICReg [14] splits the same idea into three named terms: invariance (mean squared distance between the two embeddings), variance (a hinge loss keeping the standard deviation of every dimension above a threshold, which directly forbids constant outputs) and covariance (penalizing off-diagonal covariance entries). “No negatives” does not mean “no anti-collapse design”; these methods simply put the design in the loss instead of in the architecture.

Masked modeling: learn by filling gaps

Plain version. Hide part of the input and predict what is missing. This is how BERT learned language, and it works for images too, once you choose the right amount of hiding.

BEiT [15] brought BERT-style pretraining to vision transformers. An image is split into patches; some are masked, and the network predicts discrete visual tokens for the masked positions, where the tokens come from a separately trained image tokenizer.

MAE [16] predicts raw pixels instead. It masks a large fraction of the patches, sends only the visible ones through the encoder, and reconstructs the missing pixels with a lightweight decoder. With the set of masked patch indices M\mathcal M,

Lmask=1∣M∣∑i∈M∥x^i−xi∥2,\mathcal{L}_{\mathrm{mask}} =\frac{1}{|\mathcal M|}\sum_{i\in\mathcal M} \|\hat{x}_i-x_i\|^2,

where xix_i is the true pixel vector of patch ii and x^i\hat x_i is the prediction. MAE’s design choices are deliberate. A high masking ratio (75% by default) makes the task hard enough that the network cannot simply interpolate from neighbors. The encoder only processes visible patches, which cuts pretraining cost. Normalizing each target patch by its own mean and variance improves the features. After pretraining, the decoder is discarded.

A checkerboard target, a mostly masked version, and a binary map indicating the visible positions.
Figure 7 — Masked prediction creates its own targets from the input. The checkerboard illustrates the masking operation, not semantic understanding.

Reconstruction is not the same as generation. A model that predicts masked pixels from visible ones is not trained to sample complete images from scratch. Low reconstruction loss does not guarantee the best semantic features either: background texture can dominate pixel error. Masked-modeling features are typically weaker than contrastive or distillation features under a linear probe but strong after fine-tuning, which is one reason the evaluation protocol must be stated.

Predicting features instead of pixels: JEPA

Plain version. Instead of painting in every uncertain pixel, predict a description of the hidden region.

I-JEPA [17] (Image Joint-Embedding Predictive Architecture) encodes a visible context block and predicts the encoder features of several target blocks from it. The target encoder is an EMA of the context encoder, and a predictor receives the context features plus the positions of the targets. A simplified objective is

Lpred=∥gϕ(fθ(xcontext),positions)−sg⁡(fθˉ(xtarget))∥2,\mathcal{L}_{\mathrm{pred}}= \left\|g_\phi\big(f_\theta(x_{\mathrm{context}}),\mathrm{positions}\big) -\operatorname{sg}\big(f_{\bar\theta}(x_{\mathrm{target}})\big)\right\|^2,

where fθf_\theta is the context encoder, fθˉf_{\bar\theta} the EMA target encoder, gϕg_\phi the predictor and sg⁡\operatorname{sg} stop-gradient. The formula summarizes the idea; the masking strategy (large, semantic-scale target blocks) and the architecture are part of the method, not optional details.

V-JEPA 2 [18] extends feature prediction to video at scale. Its report separates action-free pretraining on video from an action-conditioned model, trained afterward on a small amount of robot data, that is used for planning. Predicting future latent states, recognizing actions and generating photorealistic frames are different capabilities, and a “world model” claim should say which one it makes.

Language as supervision: CLIP

Plain version. Align image and caption embeddings, then compare an image with text descriptions of candidate classes. Prompt wording affects the result.

CLIP [19] trains on image–text pairs collected from the web. In a batch of NN pairs, the image encoder gives normalized embeddings u1,…,uNu_1,\ldots,u_N and the text encoder gives v1,…,vNv_1,\ldots,v_N. The N×NN\times N similarity matrix Sij=ui⊤vj/τS_{ij}=u_i^\top v_j/\tau, with a learned temperature τ\tau, has the correct pairs on its diagonal. The loss is the average of two InfoNCE losses, one over rows (which caption fits this image?) and one over columns (which image fits this caption?):

LCLIP=12[1N∑i−log⁡eSii∑jeSij+1N∑j−log⁡eSjj∑ieSij].\mathcal L_{\mathrm{CLIP}}=\frac12\Big[\frac1N\sum_{i}-\log\frac{e^{S_{ii}}}{\sum_j e^{S_{ij}}} +\frac1N\sum_{j}-\log\frac{e^{S_{jj}}}{\sum_i e^{S_{ij}}}\Big].

Zero-shot classification follows directly. Embed one prompt per class, such as “a photo of a dog”, and assign an image to the class whose prompt embedding is most similar. No task-specific labels are needed, although prompt wording affects accuracy.

SigLIP [20] replaces the softmax with an independent sigmoid loss on every image–text pair: each entry of the matrix becomes a yes/no question, “do these match?”. This removes the need to normalize over the whole batch, which simplifies large-scale distributed training.

Code · Python · 26 lines
import torch
from torch.nn import functional as F

torch.manual_seed(0)
D = 16

# --- CLIP-style symmetric loss on N matched image-text pairs
N = 6
img = F.normalize(torch.randn(N, D), dim=1)
txt = F.normalize(img + 0.3 * torch.randn(N, D), dim=1)   # caption i matches image i
logit_scale = torch.tensor(1 / 0.07)
sim = logit_scale * img @ txt.T                           # N x N, diagonal = matches
labels = torch.arange(N)
loss_clip = (F.cross_entropy(sim, labels) + F.cross_entropy(sim.T, labels)) / 2
print("CLIP loss:", round(loss_clip.item(), 3))

# zero-shot classification: embed one prompt per class, pick the most similar
class_prompts = txt[:3]                                   # stand-ins for "a photo of a {class}"
pred = (img @ class_prompts.T).argmax(1)
print("zero-shot predictions for images 0-2:", pred[:3].tolist())

# SigLIP-style alternative: an independent yes/no decision for every pair
t, b = 10.0, -10.0                                        # learnable in practice
signs = 2 * torch.eye(N) - 1                              # +1 on matches, -1 elsewhere
loss_sig = -F.logsigmoid(signs * (t * img @ txt.T + b)).sum() / N
print("sigmoid loss:", round(loss_sig.item(), 3))

The synthetic captions are noisy copies of the image embeddings, so the CLIP loss is small (0.054) and the zero-shot step recovers the right “classes” [0, 1, 2]. In real CLIP the two encoders are a vision transformer or CNN and a text transformer, and the hard part is the data: what the captions describe, and what they leave out, determines what the representation knows. The sigmoid loss value (3.473) is not comparable to the softmax loss; it uses different units and a bias term bb.

DINOv2 and DINOv3: features as a product

Plain version. With enough curated data and compute, pretraining can produce an encoder that you freeze and reuse for many different tasks, from classification to depth estimation.

DINOv2 [21] combines image-level self-distillation (as in DINO) with a patch-level masked objective, and pays close attention to data curation: building a large, diverse, deduplicated pretraining set from uncurated images by retrieval against curated ones. The result is a family of frozen encoders whose global and patch features work well with simple heads. DINOv3 [22], released in 2025, scales this further and adds Gram anchoring to counter the degradation of dense (patch) features during long training. Rather than only matching each feature independently, Gram anchoring constrains the matrix of similarities among patch features toward that of an earlier model. The report also distills a very large teacher into smaller models for deployment.

The practical question is whether frozen global or dense features solve your task. A model strong at semantic retrieval may miss a small scratch on a manufactured part. Increasing the input resolution and using patch features can help, but it changes the compute budget and the comparison protocol.

Evaluate transfer honestly

Plain version. Test features on held-out tasks or data with a stated adaptation protocol. Keep evaluation separate from feature and hyperparameter selection.

ProtocolWhat is trainedWhat it measures
kk-NNNothing; similarity to labeled reference featuresLocal structure of the representation
Linear probeLinear head; encoder frozenInformation accessible through a simple readout
Fine-tuningHead plus some or all encoder layersAdaptability under a stated label and compute budget
Dense transferSpatial head on patch mapsSuitability for pixel or region outputs

Linear probing versus fine-tuning

Fine-tuning has more freedom, so it usually wins when the target data is plentiful and similar to the pretraining data. It is not always better. Kumar et al. [23] showed theoretically and empirically that full fine-tuning can distort good pretrained features: while a randomly initialized head is still wrong, its large gradients flow into the encoder and change features in directions that suit the training distribution but hurt out-of-distribution (OOD) inputs. A linear probe cannot distort the features. Their remedy, LP-FT, is two-step: first train the linear head with the encoder frozen, then fine-tune everything starting from that head.

We reproduced the qualitative effect on the digits example. The encoder is the SimCLR CNN from above; the labeled training digits are clean; the test digits are shifted, a mild distribution shift. For each label budget, we average three random draws of the labeled examples.

Line plot of accuracy on shifted test digits versus labeled examples per class (1 to 50, log scale). Raw pixels and PCA stay near 0.3 to 0.35. The SimCLR linear probe rises from about 0.66 to 0.85. Plain fine-tuning sits lower, about 0.43 to 0.58. LP-FT is highest, about 0.68 to 0.92.
Figure 8 — Label efficiency on shifted test digits. Self-supervised features are far more label-efficient than pixels or PCA. With one fixed recipe, plain fine-tuning from a random head hurt the shifted-test accuracy, while LP-FT improved on the probe. A toy result: with a tuned learning rate or more labels, plain fine-tuning can do better.

Read Figure 8 for its shape, not its exact values. The frozen SSL features with a linear head need only a handful of labels per class to beat raw pixels with 50 labels per class. Plain fine-tuning with a randomly initialized head did worse than the frozen probe here, and LP-FT did better than both, the same ordering Kumar et al. report under distribution shift. Fine-tuning needs its own hyperparameter search, and the head initialization matters.

Rules against leakage

  • Normalize features consistently for similarity comparisons (cosine kk-NN needs normalized vectors).
  • Fit scalers, PCA and classifier hyperparameters on training and validation data only.
  • A frozen encoder still has a training history: overlap between pretraining data and evaluation data can inflate results, especially for web-scale models.
  • Report the number of labels, backbone, checkpoint, input resolution, augmentations, head, tuning budget and protocol. Choosing the best of many encoders on the test set is test leakage, even if none of them is fine-tuned.

Modern view

Surveys and review papers. Four reviews map the field from complementary angles.

  • Bengio, Courville and Vincent (TPAMI 2013) [1] frame the whole enterprise. They ask what makes one representation better than another and answer with a list of generic priors: distributed codes, depth and abstraction, disentangling of explanatory factors, manifolds and natural clustering. They review probabilistic models, autoencoders and manifold learning. It predates modern SSL but remains the best statement of why we learn representations.
  • Liu et al. (TKDE 2023) [3] classify self-supervised methods into generative (autoencoding and autoregressive reconstruction), contrastive (context–instance and instance–instance) and generative-contrastive (adversarial) groups across vision, language and graphs, and discuss why contrastive objectives were winning on vision benchmarks at the time.
  • Gui et al. (TPAMI 2024) [4] survey algorithms more recently, covering contrastive, self-distillation, redundancy-reduction and masked image modeling methods, and discuss applications and open problems such as theory, the choice of pretext task and multimodal SSL.
  • Balestriero et al. (2023) [5] is a practitioner’s guide written by many of the field’s authors. It groups methods into a deep-metric-learning family (SimCLR and relatives), a self-distillation family (BYOL, SimSiam, DINO), a canonical-correlation-analysis family (VICReg, Barlow Twins) and masked image modeling, and collects the “tricks” (augmentations, projectors, EMA schedules, batch-size effects) that the papers often bury. Prince’s textbook [24] gives a gentle, figure-rich introduction to the same ideas with notebooks.

State of practice (2023–2026). The trends visible in the papers cited above are:

  • Combined objectives. Leading image encoders mix image-level distillation with patch-level masked prediction (DINOv2 [21], DINOv3 [22]) rather than relying on one pretext task.
  • Data as a first-class design choice. DINOv2’s curation pipeline and the web-scale pair collection behind CLIP [19] and SigLIP [20] show that what goes into pretraining matters as much as the loss.
  • Dense features. Segmentation, depth and correspondence need good patch features, which motivated objectives like Gram anchoring [22].
  • Prediction in latent space and video. I-JEPA [17] and V-JEPA 2 [18] predict features instead of pixels, and extend to video and action-conditioned prediction.
  • Language-aligned encoders as the default backbone for multimodal models (see ML 10).

Open problems. We still lack a predictive theory of which pretext task suits which downstream task; collapse is prevented by recipes more than by guarantees; evaluation depends heavily on protocol and on overlap between pretraining and test data; and web-scale pretraining inherits biases and gaps from its data. Our reading of [3], [4], [5], [21], [22] is that data and target design deserve as much attention as architecture. That is an interpretation of the literature, not a theorem that one objective wins everywhere.

The next chapters change the question: instead of describing an image, how can a model describe and sample its whole distribution? See ML 06A for autoencoders and ML 07 for generative models.

Key takeaways

  • Useful features make the target task easier; distributed codes alone do not guarantee generalization.
  • Augmentations choose which differences to ignore; preserve differences needed by the downstream task.
  • Contrastive objectives distinguish positive matches from alternatives; agreement alone admits constant-output collapse.
  • Teacher asymmetry, variance constraints and redundancy penalties address collapse in different ways.
  • Reconstructing pixels, predicting features and aligning captions train different capabilities.
  • Nearest-neighbor tests, linear probes and fine-tuning answer different questions; evaluate adaptation under the expected distribution shift.

Exercises

  1. Temperature and collapse. In the first code block, change the temperature from 0.2 to 0.05 and to 1.0. Does the collapsed loss change? Does the paired loss?
Hint

The collapsed loss stays log⁡15\log 15: all non-self logits are equal, so dividing them by any τ\tau leaves the softmax uniform. The paired loss changes, because temperature rescales the differences between scores, and with real differences a smaller τ\tau gives a sharper softmax.

  1. Augmentations are assumptions. You want features for (a) counting cells in microscopy, (b) recognizing bird species, (c) reading street-sign text. For each, name one standard SimCLR augmentation you would remove and why.
Hint

(a) Random cropping can change the number of cells in view, so the “same” label is no longer the same. (b) Strong color jitter can remove plumage color, a key species cue. (c) Horizontal flips turn text into mirror text. Decide what must be preserved before choosing invariances.

  1. MAE compute. An image has 196 patches and MAE masks 75%. How many patches enter the encoder? By what factor does the number of pairwise attention scores in the encoder shrink?
Hint

49 visible patches enter the encoder. Attention scores scale with the square of the token count: (49/196)2=1/16(49/196)^2=1/16. The decoder still processes all 196 positions, but it is small.

  1. Break the digits experiment. Modify the SimCLR digits code so that augment only adds noise (no shift). Retrain and compare the shifted-test kk-NN accuracy with the original. What does this tell you?
Hint

Replace the random i, j with i, j = 1, 1. On our machine the shifted-test accuracy fell from 0.919 to 0.743, while clean accuracy stayed at 0.981. Without the shift augmentation, nothing in the objective asks for shift invariance; the robustness that remains comes from what a trained convolution-and-pooling encoder learns anyway (the untrained one in the main example had none). The augmentations, not the word “self-supervised”, decide most of what is learned.

  1. SimSiam ablation, on paper. In SimSiam, suppose you remove the stop-gradient so both branches receive gradients. Explain in two sentences why a constant output becomes an easy minimizer, and why the predictor alone is not enough to prevent it.
Hint

With gradients on both sides, the loss −cos⁡(p1,z2)-\cos(p_1,z_2) is minimized whenever p1p_1 and z2z_2 point the same way, which a constant encoder plus a predictor that outputs that same constant direction achieves for every input. The stop-gradient changes the optimization dynamics so that the target is held fixed during each step; the SimSiam paper reports rapid collapse without it.

  1. Fair probe. Design an experiment comparing a frozen pretrained encoder with PCA plus logistic regression at 10, 50 and 100 labels per class. List three things you must hold fixed and two things you must report.
Hint

Hold fixed: the labeled subsets (same examples for both), the held-out test split, and the procedure for choosing regularization (on validation data only). Report: the pretrained encoder’s training data (and any overlap with your test set) and the number of random label draws with their spread.

Paper guides

1 paper guide, each on a page of its own.

Open the paper guides →

References

  1. Y. Bengio, A. Courville and P. Vincent, “Representation Learning: A Review and New Perspectives,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, no. 8, 2013. Paper · arXiv
  2. J. Yosinski, J. Clune, Y. Bengio and H. Lipson, “How transferable are features in deep neural networks?” NeurIPS, 2014. Paper
  3. X. Liu et al., “Self-supervised Learning: Generative or Contrastive,” IEEE Transactions on Knowledge and Data Engineering, vol. 35, 2023 (online 2021). Paper · arXiv
  4. J. Gui et al., “A Survey on Self-Supervised Learning: Algorithms, Applications, and Future Trends,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024. Paper · arXiv
  5. R. Balestriero et al., “A Cookbook of Self-Supervised Learning,” arXiv:2304.12210, 2023. Paper
  6. A. van den Oord, Y. Li and O. Vinyals, “Representation Learning with Contrastive Predictive Coding,” arXiv:1807.03748, 2018. Paper
  7. T. Chen, S. Kornblith, M. Norouzi and G. Hinton, “A Simple Framework for Contrastive Learning of Visual Representations,” ICML, 2020. Paper
  8. K. He, H. Fan, Y. Wu, S. Xie and R. Girshick, “Momentum Contrast for Unsupervised Visual Representation Learning,” CVPR, 2020. Paper
  9. T. Wang and P. Isola, “Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere,” ICML, 2020. Paper
  10. J.-B. Grill et al., “Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning,” NeurIPS, 2020. Paper
  11. X. Chen and K. He, “Exploring Simple Siamese Representation Learning,” CVPR, 2021. Paper
  12. M. Caron et al., “Emerging Properties in Self-Supervised Vision Transformers,” ICCV, 2021. Paper
  13. J. Zbontar, L. Jing, I. Misra, Y. LeCun and S. Deny, “Barlow Twins: Self-Supervised Learning via Redundancy Reduction,” ICML, 2021. Paper
  14. A. Bardes, J. Ponce and Y. LeCun, “VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning,” ICLR, 2022. Paper
  15. H. Bao, L. Dong, S. Piao and F. Wei, “BEiT: BERT Pre-Training of Image Transformers,” ICLR, 2022. Paper
  16. K. He et al., “Masked Autoencoders Are Scalable Vision Learners,” CVPR, 2022. Paper
  17. M. Assran et al., “Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture,” CVPR, 2023. Paper
  18. M. Assran et al., “V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning,” technical report, 2025. Report
  19. A. Radford et al., “Learning Transferable Visual Models From Natural Language Supervision,” ICML, 2021. Paper
  20. X. Zhai, B. Mustafa, A. Kolesnikov and L. Beyer, “Sigmoid Loss for Language Image Pre-Training,” ICCV, 2023. Paper
  21. M. Oquab et al., “DINOv2: Learning Robust Visual Features without Supervision,” Transactions on Machine Learning Research, 2024. Paper
  22. O. Siméoni et al., “DINOv3,” technical report, 2025. Report
  23. A. Kumar, A. Raghunathan, R. Jones, T. Ma and P. Liang, “Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution,” ICLR, 2022. Paper
  24. S. J. D. Prince, Understanding Deep Learning, MIT Press, 2023. Book and notebooks

Check your understanding

This tutorial has 6 exercises. Can you solve them without opening the hints?

Go to the exercises ↑

Next ML 06A · Linear Factor Models and Autoencoders · 28 min