Full search page →

ML 08 · Diffusion Models and Flow Matching

Machine LearningAdvanced44 minUpdated Nov 6, 2026

Needs: ML 07 · Generative Models: Autoencoders, VAEs, GANs and Flows, ML 05 · Attention and Vision Transformers

Navigate: this page and this track: Navigate: this page and this track

On this page

Paper guides · 6

  1. DDPM — noise prediction and the quality–likelihood tradeoff
  2. Flow Matching — regress a velocity, then integrate it
  3. MeanFlow — average velocity over an interval
  4. Improved MeanFlow — prediction inputs and flexible guidance
  5. JiT — clean-image prediction in pixel space
  6. Pixel MeanFlow — one-step pixels through a denoised field

Machine Learning

View the whole course tree →
Explore this chapter 2 interactive experiments

Mix the astronaut photograph with fixed noise, look at the noise, clean-image and velocity targets side by side, then give the network a small error and see which conversion back to the clean image survives.

What you’ll learn

  • Build a forward noise process and a reverse prediction task.
  • Relate variational training to noise, clean-image and velocity targets.
  • Separate model predictions from numerical sampling methods.
  • Compare conditioning, generative representations and flow matching.
  • Evaluate complete image/video pipelines and few-step methods.

The big picture

A diffusion model learns to undo controlled corruption. During training we take a real example, add a known amount of Gaussian noise, and ask a network to tell us what noise we added. Because we added the noise ourselves, the correct answer is always available: no labels, no adversary, just regression. During generation we start from pure noise and apply the learned “denoise a little” step many times until an example appears.

Why does this matter? Since 2021, diffusion-style models have produced many of the state-of-the-art results in image, video and audio synthesis, and the same machinery is used for inpainting, super-resolution, image editing, 3-D generation and scientific data. Diffusion also replaced two pain points of earlier families covered in ML 07: GANs need a fragile two-player game, and VAEs tend to produce blurry samples. Diffusion training is a stable regression, and its samples are sharp because generation is spread over many small, easy steps.

The price is sampling cost: a classic sampler calls the network hundreds of times per image. Much of the research since 2021 — DDIM, fast ODE solvers, distillation, consistency models, flow matching, rectified flow — is about keeping the quality while cutting the number of network calls.

The chapter follows the organization of the major surveys. Yang et al. [1] group the field into efficient sampling, improved likelihood and data with special structure; Croitoru et al. [27] group it by formulation (DDPM, noise-conditioned score networks, SDEs); Lipman et al.’s flow-matching guide [2] and the monograph by Lai et al. [29] present diffusion and flows as one family of learned transport. Prince’s textbook [14] is a gentle companion for the derivations. Throughout, we use one explicit time convention per section and say when it changes — mixing conventions is the most common source of sign errors in this area.

Forward diffusion has a closed form

Plain version. The forward process gradually weakens the image signal and adds Gaussian noise. A suitable terminal schedule makes the distribution approximately standard Gaussian.

The idea goes back to Sohl-Dickstein et al. [15], who proposed destroying structure with a slow forward diffusion and learning to reverse it; DDPM [3] made it practical. Let x0∈Rdx_0\in\mathbb R^d be a data example (an image flattened into dd numbers, scaled to roughly [−1,1][-1,1]). Pick a noise schedule β1,…,βT\beta_1,\ldots,\beta_T with 0<βt<10\lt\beta_t\lt1 and write αt=1−βt\alpha_t=1-\beta_t. The forward (noising) process is a Markov chain

q(xt∣xt−1)=N ⁣(αt xt−1, βtI),t=1,…,T,q(x_t\mid x_{t-1})=\mathcal N\!\left(\sqrt{\alpha_t}\,x_{t-1},\ \beta_t I\right),\qquad t=1,\ldots,T,

where N(μ,Σ)\mathcal N(\mu,\Sigma) is a Gaussian with mean μ\mu and covariance Σ\Sigma, and II is the d×dd\times d identity. The factor αt\sqrt{\alpha_t} shrinks the signal slightly so that the total variance stays bounded instead of growing; this is why this family is called variance preserving (VP).

Deriving the one-shot formula

Write one step as xt=αtxt−1+1−αt ϵtx_t=\sqrt{\alpha_t}x_{t-1}+\sqrt{1-\alpha_t}\,\epsilon_t with ϵt∼N(0,I)\epsilon_t\sim\mathcal N(0,I). Substituting the previous step,

xt=αt(αt−1xt−2+1−αt−1 ϵt−1)+1−αt ϵt=αtαt−1 xt−2+αt(1−αt−1) ϵt−1+1−αt ϵt⏟sum of independent Gaussians.\begin{aligned} x_t&=\sqrt{\alpha_t}\left(\sqrt{\alpha_{t-1}}x_{t-2}+\sqrt{1-\alpha_{t-1}}\,\epsilon_{t-1}\right)+\sqrt{1-\alpha_t}\,\epsilon_t\\ &=\sqrt{\alpha_t\alpha_{t-1}}\,x_{t-2}+\underbrace{\sqrt{\alpha_t(1-\alpha_{t-1})}\,\epsilon_{t-1}+\sqrt{1-\alpha_t}\,\epsilon_t}_{\text{sum of independent Gaussians}}. \end{aligned}

The two noise terms are independent zero-mean Gaussians, so their sum is Gaussian with variance αt(1−αt−1)+(1−αt)=1−αtαt−1\alpha_t(1-\alpha_{t-1})+(1-\alpha_t)=1-\alpha_t\alpha_{t-1}. Repeating the argument down to x0x_0 gives, with αˉt=∏s=1tαs\bar\alpha_t=\prod_{s=1}^t\alpha_s,

q(xt∣x0)=N ⁣(αˉt x0, (1−αˉt)I)⟺xt=αˉt x0+1−αˉt ϵ,ϵ∼N(0,I).q(x_t\mid x_0)=\mathcal N\!\left(\sqrt{\bar\alpha_t}\,x_0,\ (1-\bar\alpha_t)I\right) \quad\Longleftrightarrow\quad x_t=\sqrt{\bar\alpha_t}\,x_0+\sqrt{1-\bar\alpha_t}\,\epsilon,\qquad \epsilon\sim\mathcal N(0,I).

Here αˉt\bar\alpha_t is the fraction of signal variance that survives to step tt, and ϵ\epsilon is a single standard Gaussian vector. This formula is what makes training cheap: we can jump straight to any noise level without simulating the earlier steps.

A synthetic rectangular image corrupted with a fixed Gaussian noise realization at four increasing noise strengths.
Figure 1 — The closed-form forward process at several signal levels. A shared noise realization makes the effect visible; training draws fresh noise for every example and every step.

Noise schedules and the signal-to-noise ratio

The signal-to-noise ratio at step tt is

SNR(t)=αˉt1−αˉt,\mathrm{SNR}(t)=\frac{\bar\alpha_t}{1-\bar\alpha_t},

the ratio of signal variance to noise variance (for unit-variance data). It decreases monotonically from very large at t=1t=1 to nearly zero at t=Tt=T. Two requirements constrain the schedule: αˉT\bar\alpha_T must be close to 00, so that xTx_T is practically a standard Gaussian (the sampler starts from N(0,I)\mathcal N(0,I)); and the steps in between should spend the training budget on noise levels where the network has something useful to learn.

DDPM [3] used T=1000T=1000 and a linear schedule βt\beta_t from 10−410^{-4} to 0.020.02. Nichol and Dhariwal [19] observed that this destroys information quickly: for a large part of the chain the image is already nearly pure noise. Their cosine schedule defines αˉt=f(t)/f(0)\bar\alpha_t=f(t)/f(0) with f(t)=cos⁡2 ⁣(t/T+s1+s⋅π2)f(t)=\cos^2\!\big(\tfrac{t/T+s}{1+s}\cdot\tfrac{\pi}{2}\big) and a small offset s=0.008s=0.008, and clips each βt\beta_t at 0.9990.999. Figure 2 compares the two. Karras et al. [20] later argued that the schedule, the network’s input/output scaling and the distribution of training noise levels are separate design choices that are best tuned independently.

Two line plots over 1000 steps comparing the linear and cosine noise schedules: cumulative signal fraction and log signal-to-noise ratio.
Figure 2 — Linear versus cosine schedule for T = 1000. The cosine schedule keeps more signal for longer and spreads the log-SNR more evenly over the steps.

The reverse process and the variational bound

Plain version. To generate, we need the reverse step: “given a noisy example, what did the slightly less noisy example look like?” We cannot compute this directly for real data, but we can compute it if we also know the clean example — and that turns out to give a simple training loss.

The model and the bound

The generative model is a reverse Markov chain that starts at p(xT)=N(0,I)p(x_T)=\mathcal N(0,I) and uses learned Gaussian transitions

pθ(xt−1∣xt)=N ⁣(μθ(xt,t), σt2I),p_\theta(x_{t-1}\mid x_t)=\mathcal N\!\left(\mu_\theta(x_t,t),\ \sigma_t^2 I\right),

where μθ\mu_\theta is a neural network (with parameters θ\theta) and σt2\sigma_t^2 is a variance we will choose. This is a latent-variable model with latents x1:Tx_{1:T}, exactly like a deep VAE whose encoder qq is fixed rather than learned (see ML 07). The same Jensen-inequality argument gives an evidence lower bound. Following [3], [15], the negative bound can be rearranged into a sum of KL divergences:

−log⁡pθ(x0) ≤ Eq[DKL ⁣(q(xT∣x0) ∥ p(xT))⏟LT+∑t=2TDKL ⁣(q(xt−1∣xt,x0) ∥ pθ(xt−1∣xt))⏟Lt−1−log⁡pθ(x0∣x1)⏟L0].-\log p_\theta(x_0)\ \le\ \mathbb E_q\Big[ \underbrace{D_{\mathrm{KL}}\!\big(q(x_T\mid x_0)\,\|\,p(x_T)\big)}_{L_T} +\sum_{t=2}^{T}\underbrace{D_{\mathrm{KL}}\!\big(q(x_{t-1}\mid x_t,x_0)\,\|\,p_\theta(x_{t-1}\mid x_t)\big)}_{L_{t-1}} \underbrace{-\log p_\theta(x_0\mid x_1)}_{L_0}\Big].

DKLD_{\mathrm{KL}} is the Kullback–Leibler divergence. LTL_T has no learnable parameters (it is small when αˉT≈0\bar\alpha_T\approx0); L0L_0 is a decoder term for the last step; and each Lt−1L_{t-1} compares the model’s reverse step with the forward posterior q(xt−1∣xt,x0)q(x_{t-1}\mid x_t,x_0) — the reverse step that would be exact if the clean example x0x_0 were known.

The forward posterior is Gaussian

By Bayes’ rule, q(xt−1∣xt,x0)∝q(xt∣xt−1) q(xt−1∣x0)q(x_{t-1}\mid x_t,x_0)\propto q(x_t\mid x_{t-1})\,q(x_{t-1}\mid x_0). Both factors are Gaussian in xt−1x_{t-1}, so their product is Gaussian. Collecting the quadratic and linear terms in xt−1x_{t-1} (completing the square) gives

q(xt−1∣xt,x0)=N ⁣(μ~t(xt,x0), β~tI),q(x_{t-1}\mid x_t,x_0)=\mathcal N\!\left(\tilde\mu_t(x_t,x_0),\ \tilde\beta_t I\right), μ~t=αˉt−1 βt1−αˉt x0+αt (1−αˉt−1)1−αˉt xt,β~t=1−αˉt−11−αˉt βt.\tilde\mu_t=\frac{\sqrt{\bar\alpha_{t-1}}\,\beta_t}{1-\bar\alpha_t}\,x_0+\frac{\sqrt{\alpha_t}\,(1-\bar\alpha_{t-1})}{1-\bar\alpha_t}\,x_t, \qquad \tilde\beta_t=\frac{1-\bar\alpha_{t-1}}{1-\bar\alpha_t}\,\beta_t .

The posterior mean is a weighted average of the clean example and the current noisy one; the posterior variance β~t\tilde\beta_t is slightly smaller than βt\beta_t. You do not have to trust the algebra: the snippet below samples the forward chain and fits a regression of xt−1x_{t-1} on xtx_t.

Code · Python · 23 lines
import numpy as np

rng = np.random.default_rng(0)
beta = np.linspace(1e-4, 0.02, 1000)
alpha = 1 - beta
abar = np.cumprod(alpha)
t, x0, n = 300, 1.3, 400_000           # check q(x_{t-1} | x_t, x_0) at step t

# Sample the forward chain: x_{t-1} from q(x_{t-1}|x_0), then one more step.
x_prev = np.sqrt(abar[t-1])*x0 + np.sqrt(1-abar[t-1])*rng.normal(size=n)
x_t = np.sqrt(alpha[t])*x_prev + np.sqrt(beta[t])*rng.normal(size=n)

# Empirical regression of x_{t-1} on x_t (the posterior mean is linear in x_t).
slope, intercept = np.polyfit(x_t, x_prev, 1)
resid_var = np.var(x_prev - (slope*x_t + intercept))

# Closed-form posterior from the derivation.
c_x0 = np.sqrt(abar[t-1])*beta[t] / (1-abar[t])
c_xt = np.sqrt(alpha[t])*(1-abar[t-1]) / (1-abar[t])
var = beta[t]*(1-abar[t-1]) / (1-abar[t])
print(f"slope     {slope:.4f}  vs  {c_xt:.4f}")
print(f"intercept {intercept:.4f}  vs  {c_x0*x0:.4f}")
print(f"variance  {resid_var:.6f}  vs  {var:.6f}")

The output matches to three or four decimals (slope 0.9934 vs 0.9930, variance 0.006050 vs 0.006052); the small gaps are Monte Carlo error.

From the bound to the simplified loss

Now use the forward formula backwards: x0=(xt−1−αˉt ϵ)/αˉtx_0=(x_t-\sqrt{1-\bar\alpha_t}\,\epsilon)/\sqrt{\bar\alpha_t}. Substituting into μ~t\tilde\mu_t and simplifying with αˉt=αtαˉt−1\bar\alpha_t=\alpha_t\bar\alpha_{t-1},

μ~t=1αt(xt−βt1−αˉt ϵ).\tilde\mu_t=\frac{1}{\sqrt{\alpha_t}}\left(x_t-\frac{\beta_t}{\sqrt{1-\bar\alpha_t}}\,\epsilon\right).

At sampling time the network sees xtx_t but not ϵ\epsilon. So DDPM parameterizes the model mean in exactly the same form, with the unknown noise replaced by a prediction ϵθ(xt,t)\epsilon_\theta(x_t,t):

μθ(xt,t)=1αt(xt−βt1−αˉt ϵθ(xt,t)).\mu_\theta(x_t,t)=\frac1{\sqrt{\alpha_t}}\left(x_t-\frac{\beta_t}{\sqrt{1-\bar\alpha_t}}\,\epsilon_\theta(x_t,t)\right).

The KL between two Gaussians with the same fixed covariance σt2I\sigma_t^2I is a scaled squared distance between their means, so

Lt−1=12σt2∥μ~t−μθ∥2+C=βt22σt2 αt (1−αˉt) ∥ϵ−ϵθ(xt,t)∥2+C,L_{t-1}=\frac{1}{2\sigma_t^2}\big\|\tilde\mu_t-\mu_\theta\big\|^2+C =\frac{\beta_t^2}{2\sigma_t^2\,\alpha_t\,(1-\bar\alpha_t)}\,\big\|\epsilon-\epsilon_\theta(x_t,t)\big\|^2+C,

where CC does not depend on θ\theta. Every term of the bound is therefore a weighted noise-prediction error. Ho et al. [3] found that dropping the weight, and sampling tt uniformly, trains better:

Lsimple=Ex0, t∼U{1,…,T}, ϵ∼N(0,I)∥ϵ−ϵθ ⁣(αˉtx0+1−αˉt ϵ, t)∥2.\mathcal L_{\mathrm{simple}} =\mathbb E_{x_0,\ t\sim\mathcal U\{1,\ldots,T\},\ \epsilon\sim\mathcal N(0,I)} \Big\|\epsilon-\epsilon_\theta\!\big(\sqrt{\bar\alpha_t}x_0+\sqrt{1-\bar\alpha_t}\,\epsilon,\ t\big)\Big\|^2 .

Read it as an algorithm: pick a training example, pick a random step, pick random noise, build xtx_t in one line, and regress the noise. The time tt is fed to the network (usually through a sinusoidal embedding) because the same pixel values mean different things at different noise levels.

Two points are easy to get wrong. First, Lsimple\mathcal L_{\mathrm{simple}} is not the likelihood bound: removing the weights relatively down-weights the low-noise steps, which helps sample quality but hurts the reported log-likelihood. Improved DDPM [19] recovered good likelihoods by learning the variances σt2\sigma_t^2 (as an interpolation between βt\beta_t and β~t\tilde\beta_t) with a hybrid loss. Second, the same network also yields a clean estimate,

x^0(xt,t)=xt−1−αˉt ϵθ(xt,t)αˉt,\hat x_0(x_t,t)=\frac{x_t-\sqrt{1-\bar\alpha_t}\,\epsilon_\theta(x_t,t)}{\sqrt{\bar\alpha_t}},

which every modern sampler uses internally. When αˉt\bar\alpha_t is tiny, the division amplifies prediction errors, so x^0\hat x_0 at very high noise is a blurry average, not a sample.

Optional: DDPM (in the Paper guides). Read the original figure, evidence, design choices and limitations.

Noise, clean data, velocity and the score

Plain version. Predicting the noise, predicting the clean image and predicting “which direction makes this example more typical” are three ways to express the same knowledge. They differ in scaling, and scaling changes how the training loss weights different noise levels.

Three parameterizations

Because xt=αˉtx0+1−αˉtϵx_t=\sqrt{\bar\alpha_t}x_0+\sqrt{1-\bar\alpha_t}\epsilon is a fixed linear relation, a network can output any one of

  • the noise ϵ\epsilon (DDPM [3]),
  • the clean example x0x_0,
  • the velocity v=αˉt ϵ−1−αˉt x0v=\sqrt{\bar\alpha_t}\,\epsilon-\sqrt{1-\bar\alpha_t}\,x_0, introduced by Salimans and Ho [22] for distillation, which stays well scaled at both ends of the schedule.

Given xtx_t and tt, each converts to the others exactly. However, an unweighted MSE on ϵ\epsilon, on x0x_0 or on vv weights the noise levels differently, so the choice affects training. Karras et al. [20] made this explicit with input/output “preconditioning” coefficients that keep the network’s inputs and targets at unit variance for every noise level. Note that this diffusion “velocity” is a reparameterization of signal and noise; it is related to, but not identical with, the transport velocity of flow matching later in this chapter.

Exact conversion does not make finite networks equivalent

For the variance-preserving path, write a=αˉta=\sqrt{\bar\alpha_t} and b=1−αˉtb=\sqrt{1-\bar\alpha_t}. Thus a2+b2=1a^2+b^2=1 and xt=ax0+bϵx_t=ax_0+b\epsilon. The diffusion velocity vVP=aϵ−bx0v_{\mathrm{VP}}=a\epsilon-bx_0 completes a rotation:

x0=axt−bvVP,ϵ=bxt+avVP,x^0(ϵ)=(xt−bϵ^)/a,ϵ^(x)=(xt−ax^0)/b.\begin{aligned} x_0&=a x_t-bv_{\mathrm{VP}}, &\epsilon&=b x_t+a v_{\mathrm{VP}},\\ \hat x_0^{(\epsilon)}&=(x_t-b\hat\epsilon)/a, & \hat\epsilon^{(x)}&=(x_t-a\hat x_0)/b. \end{aligned}

Here a,ba,b are known schedule coefficients; vVPv_{\mathrm{VP}} is the target used in [22]; hats denote estimates; and superscripts identify the network’s direct output. The last two conversions require nonzero divisors. Their instability near an endpoint is a numerical and statistical issue, even though the algebra is exact in the interior.

A noise error δϵ\delta_\epsilon becomes a clean-image error −(b/a)δϵ-(b/a)\delta_\epsilon. Therefore, for corresponding predictions at a fixed noise level,

∥x^0−x0∥2=b2a2∥ϵ^−ϵ∥2=1SNR(t)∥ϵ^−ϵ∥2,∥x^0(v)−x0∥2=b2∥v^VP−vVP∥2.\|\hat x_0-x_0\|^2=\frac{b^2}{a^2}\|\hat\epsilon-\epsilon\|^2 =\frac{1}{\mathrm{SNR}(t)}\|\hat\epsilon-\epsilon\|^2, \qquad \|\hat x_0^{(v)}-x_0\|^2=b^2\|\hat v_{\mathrm{VP}}-v_{\mathrm{VP}}\|^2.

SNR(t)=a2/b2\mathrm{SNR}(t)=a^2/b^2 is the signal-to-noise ratio; the second identity follows from the rotation above. Equal unweighted losses consequently emphasize different times. Changing the target without recording time sampling and loss weighting confounds an experiment [20], [22].

There is also a capacity issue. If a network directly outputs a clean image, its final representation can discard nuisance noise. A network directly outputting noise may need to preserve more coordinates of the noisy input, or use a skip connection that supplies those coordinates. Converting one network’s output afterward does not make its architecture identical to a network trained to output another target. This distinction becomes especially relevant for large pixel patches and high-dimensional latents; the JiT discussion below supplies recent evidence [42]. It does not imply that noise prediction generally fails: the earlier DDPM examples and dense image denoisers are counterexamples.

Code · Python · 28 lines
import numpy as np

rng = np.random.default_rng(7)
x0 = rng.normal(size=32)
eps = rng.normal(size=32)
a, b = np.sqrt(.2), np.sqrt(.8)
xt = a*x0 + b*eps
v_vp = a*eps - b*x0
assert np.allclose(a*xt - b*v_vp, x0)
assert np.allclose(b*xt + a*v_vp, eps)

# A deliberately imperfect noise estimate: compare equivalent losses.
eps_hat = eps + .03*rng.normal(size=32)
x_hat = (xt - b*eps_hat)/a
noise_mse = np.mean((eps_hat - eps)**2)
clean_mse = np.mean((x_hat - x0)**2)
assert np.allclose(clean_mse/noise_mse, b*b/(a*a))
print("Clean/noise MSE ratio:", round(clean_mse/noise_mse, 3))

# A straight flow uses a different path and a different velocity.
t = .35
zt = (1-t)*eps + t*x0
u = x0 - eps
x_from_flow = zt + (1-t)*u
u_from_clean = (x0 - zt)/(1-t)
assert np.allclose(x_from_flow, x0)
assert np.allclose(u_from_clean, u)
print("VP velocity equals straight-flow target:", np.allclose(v_vp, u))

The example prints a loss ratio of 4.0 and False for the two velocity targets. Both use the letter vv in parts of the literature; a checkpoint’s scheduler must identify the actual path, normalization and time direction.

What is a score, and why does denoising estimate it?

The score of a density pp is ∇xlog⁡p(x)\nabla_x\log p(x): a vector field that points toward higher probability. If you know the score, you can sample with Langevin dynamics: repeatedly step a little along the score and add a little noise. Song and Ermon [17] trained “noise-conditional score networks” at many noise levels and sampled with annealed Langevin dynamics — a parallel route to diffusion that turned out to be the same idea.

The bridge is denoising score matching (Vincent [16]). For Gaussian corruption, the best possible noise predictor (in mean squared error) is a conditional expectation, ϵ∗(xt,t)=E[ϵ∣xt]\epsilon^*(x_t,t)=\mathbb E[\epsilon\mid x_t], and Tweedie’s formula [40] links it to the score of the noisy marginal ptp_t:

∇xtlog⁡pt(xt)=−E[ϵ∣xt]1−αˉt⟹sθ(xt,t)≈−ϵθ(xt,t)1−αˉt,\nabla_{x_t}\log p_t(x_t)=-\frac{\mathbb E[\epsilon\mid x_t]}{\sqrt{1-\bar\alpha_t}} \quad\Longrightarrow\quad s_\theta(x_t,t)\approx-\frac{\epsilon_\theta(x_t,t)}{\sqrt{1-\bar\alpha_t}},

with sθs_\theta the learned score. The same formula in “clean” units reads E[x0∣xt]=(xt+(1−αˉt)∇log⁡pt(xt))/αˉt\mathbb E[x_0\mid x_t]=\big(x_t+(1-\bar\alpha_t)\nabla\log p_t(x_t)\big)/\sqrt{\bar\alpha_t}. Two warnings: this is the score of the noisy marginal distribution at level tt, not the gradient for any single training image; and it holds for the optimal predictor — a trained network approximates it.

You can verify Tweedie’s formula where everything is known: one-dimensional Gaussian data, for which the best noise predictor is linear in xtx_t.

Code · Python · 16 lines
import numpy as np

rng = np.random.default_rng(0)
s, abar, n = 0.7, 0.4, 500_000     # data x0 ~ N(0, s^2); one noise level
x0 = s*rng.normal(size=n)
eps = rng.normal(size=n)
xt = np.sqrt(abar)*x0 + np.sqrt(1-abar)*eps

# "Train" the best linear noise predictor eps_hat = w * x_t by least squares.
w = (xt @ eps) / (xt @ xt)
score_from_eps = -w / np.sqrt(1-abar)       # s(x_t) = -eps_hat / sqrt(1-abar)

# Exact: x_t ~ N(0, abar s^2 + 1 - abar), so grad log p_t(x) = -x / var.
exact = -1 / (abar*s**2 + 1 - abar)
print(f"score slope from noise predictor: {score_from_eps:.4f}")
print(f"exact score slope:                {exact:.4f}")

It prints −1.2560-1.2560 versus −1.2563-1.2563: a regression on noise recovers the score.

The SDE view

Song et al. [4] took the step count to infinity. With continuous time t∈[0,1]t\in[0,1] and a rate β(t)\beta(t), the VP forward process becomes the stochastic differential equation (SDE)

dx=−12β(t) x dt+β(t) dw,dx=-\tfrac12\beta(t)\,x\,dt+\sqrt{\beta(t)}\,dw,

where ww is a standard Wiener process (Brownian motion). A classical result says that this process run backwards in time is again an SDE, and that the only unknown in it is the score:

dx=[−12β(t) x−β(t) ∇xlog⁡pt(x)]dt+β(t) dwˉ,dx=\Big[-\tfrac12\beta(t)\,x-\beta(t)\,\nabla_x\log p_t(x)\Big]dt+\sqrt{\beta(t)}\,d\bar w,

with dtdt negative and wˉ\bar w a reverse-time Wiener process. Moreover, there is a deterministic probability-flow ODE whose solutions have the same marginal distributions ptp_t at every time:

dxdt=−12β(t)[x+∇xlog⁡pt(x)].\frac{dx}{dt}=-\tfrac12\beta(t)\Big[x+\nabla_x\log p_t(x)\Big].

This unifies the field. DDPM’s ancestral sampler relates to a discretization of the reverse SDE; NCSN [17] corresponds to a “variance exploding” SDE. The probability-flow ODE defines a continuous normalizing flow whose log-density follows an instantaneous change-of-variables identity [4]. The identity is exact; a computed likelihood still has ODE integration error and, when the divergence is estimated stochastically, estimator error. With an approximate learned score, the ODE and reverse SDE need not have identical marginals. The ODE perspective also motivates deterministic sampling and general-purpose numerical solvers.

Sampling is a reverse process

Plain version. A loss evaluated at one random noise level is not a generator. To generate, you must chain many reverse steps with a consistent schedule, starting from pure noise — and you can choose how many steps and how much fresh randomness to inject.

DDPM ancestral sampling

DDPM samples by following the learned reverse chain:

xt−1=μθ(xt,t)+σtz,z∼N(0,I),σt2∈{βt, β~t},x_{t-1}=\mu_\theta(x_t,t)+\sigma_t z,\qquad z\sim\mathcal N(0,I),\qquad \sigma_t^2\in\{\beta_t,\ \tilde\beta_t\},

with no noise added at the final step. Ho et al. [3] report similar results for σt2=βt\sigma_t^2=\beta_t and σt2=β~t\sigma_t^2=\tilde\beta_t; learned variances [19] help most when using fewer steps.

Code: DDPM on a two-dimensional distribution

This complete CPU example (about 10 seconds) uses two Gaussian clusters centered at x=±1.5x=\pm1.5 with standard deviation 0.250.25. It contains every operation of an image model — schedule, random-time training, noise prediction, reverse mean — without an image-scale budget. With 100 steps and this schedule, almost no signal survives to t=Tt=T.

Code · Python · 41 lines
import torch
from torch import nn
from torch.nn import functional as F

torch.manual_seed(0)
torch.set_num_threads(1)
T = 100
beta = torch.linspace(.001, .15, T)
alpha = 1-beta
abar = alpha.cumprod(0)
previous = torch.cat([torch.ones(1), abar[:-1]])
post_var = beta * (1-previous) / (1-abar)
net = nn.Sequential(nn.Linear(3, 64), nn.SiLU(),
                    nn.Linear(64, 64), nn.SiLU(), nn.Linear(64, 2))
opt = torch.optim.Adam(net.parameters(), lr=.002)
def data(n):
    side = torch.randint(2, (n, 1))*2-1
    return torch.cat([side*1.5, torch.zeros(n, 1)], 1) + .25*torch.randn(n, 2)
for step in range(3000):
    x0 = data(128)
    t = torch.randint(T, (len(x0),))
    eps = torch.randn_like(x0)
    a = abar[t, None]
    xt = a.sqrt()*x0 + (1-a).sqrt()*eps
    predicted = net(torch.cat([xt, t[:, None].float()/(T-1)], 1))
    loss = F.mse_loss(predicted, eps)
    opt.zero_grad(); loss.backward(); opt.step()
net.eval()
with torch.no_grad():
    x = torch.randn(512, 2)
    for t in reversed(range(T)):
        time = torch.full((len(x), 1), t/(T-1))
        eps = net(torch.cat([x, time], 1))
        mean = (x-beta[t]/(1-abar[t]).sqrt()*eps)/alpha[t].sqrt()
        x = mean + post_var[t].sqrt()*torch.randn_like(x) if t else mean
    assert torch.isfinite(x).all()
    print("terminal signal:", round(abar[-1].item(), 6))
    left = x[x[:, 0] < 0]
    print("fraction in left cluster:", round((x[:, 0] < 0).float().mean().item(), 3))
    print("left cluster: mean x", round(left[:, 0].mean().item(), 2),
          "std", round(left[:, 0].std().item(), 2), "(data: -1.5 and 0.25)")

On our machine it prints a terminal signal of 0.000349, a left-cluster fraction of 0.441, and a left-cluster mean and standard deviation of −1.47-1.47 and 0.250.25. The cluster shapes are right; the 44/56 split shows the small network has not learned the mode weights exactly. This is why a single summary number is never enough: always plot samples, and check mode coverage, within-mode spread and outliers against fresh data.

DDIM: same training, different sampler

Song, Meng and Ermon [5] noticed that Lsimple\mathcal L_{\mathrm{simple}} only depends on the marginals q(xt∣x0)q(x_t\mid x_0), not on the Markov chain that produced them. Many non-Markovian forward processes share those marginals, and each gives a different reverse sampler — all usable with the same trained ϵθ\epsilon_\theta. The DDIM update, for any decreasing subsequence of steps τ1>τ2>⋯\tau_1>\tau_2>\cdots, is

xτi+1=αˉτi+1  x^0+1−αˉτi+1−σ2  ϵθ(xτi,τi)+σz,σ=η1−αˉτi+11−αˉτi(1−αˉτiαˉτi+1),x_{\tau_{i+1}}=\sqrt{\bar\alpha_{\tau_{i+1}}}\;\hat x_0 +\sqrt{1-\bar\alpha_{\tau_{i+1}}-\sigma^2}\;\epsilon_\theta(x_{\tau_i},\tau_i) +\sigma z, \qquad \sigma=\eta\sqrt{\frac{1-\bar\alpha_{\tau_{i+1}}}{1-\bar\alpha_{\tau_i}}\Big(1-\frac{\bar\alpha_{\tau_i}}{\bar\alpha_{\tau_{i+1}}}\Big)} ,

where x^0\hat x_0 is the clean estimate at xτix_{\tau_i} and η≥0\eta\ge0 controls stochasticity. Read it as: “estimate the clean example, then re-noise it to the next, lower noise level using the predicted noise direction.” With η=1\eta=1 and consecutive steps, this recovers DDPM’s posterior variance; with η=0\eta=0 the sampler is deterministic and is a discretization of the probability-flow ODE. Determinism has practical consequences: the same starting noise always gives the same output, so you can interpolate in noise space or invert a real image to its noise (DDIM inversion), which is the basis of many editing methods.

The following standalone example repeats the same small training setup so it can run independently. When experimenting in one session, reuse the already trained net, abar and T instead:

Code · Python · 50 lines
import torch
from torch import nn
from torch.nn import functional as F

torch.manual_seed(0)
torch.set_num_threads(1)
T = 100
beta = torch.linspace(.001, .15, T)
alpha = 1-beta
abar = alpha.cumprod(0)
previous = torch.cat([torch.ones(1), abar[:-1]])
post_var = beta * (1-previous) / (1-abar)
net = nn.Sequential(nn.Linear(3, 64), nn.SiLU(),
                    nn.Linear(64, 64), nn.SiLU(), nn.Linear(64, 2))
opt = torch.optim.Adam(net.parameters(), lr=.002)
def data(n):
    side = torch.randint(2, (n, 1))*2-1
    return torch.cat([side*1.5, torch.zeros(n, 1)], 1) + .25*torch.randn(n, 2)
for step in range(3000):
    x0 = data(128)
    t = torch.randint(T, (len(x0),))
    eps = torch.randn_like(x0)
    a = abar[t, None]
    xt = a.sqrt()*x0 + (1-a).sqrt()*eps
    predicted = net(torch.cat([xt, t[:, None].float()/(T-1)], 1))
    loss = F.mse_loss(predicted, eps)
    opt.zero_grad(); loss.backward(); opt.step()
net.eval()

def ddim_sample(n, steps, eta=0.0, seed=1):
    g = torch.Generator().manual_seed(seed)
    taus = torch.linspace(T-1, 0, steps).round().long()
    x = torch.randn(n, 2, generator=g)
    with torch.no_grad():
        for i, t in enumerate(taus):
            time = torch.full((n, 1), t.item()/(T-1))
            eps = net(torch.cat([x, time], 1))
            a_t = abar[t]
            a_prev = abar[taus[i+1]] if i+1 < steps else torch.tensor(1.)
            x0_hat = (x - (1-a_t).sqrt()*eps) / a_t.sqrt()
            sigma = eta*((1-a_prev)/(1-a_t)*(1-a_t/a_prev)).sqrt()
            x = (a_prev.sqrt()*x0_hat + (1-a_prev-sigma**2).sqrt()*eps
                 + sigma*torch.randn(n, 2, generator=g))
    return x

for steps in [5, 20, 100]:
    s = ddim_sample(2000, steps)
    left = s[s[:, 0] < 0]
    print(f"{steps:3d} steps: left fraction {(s[:, 0] < 0).float().mean():.3f}, "
          f"left-cluster mean x {left[:, 0].mean():.2f}, std {left[:, 0].std():.2f}")

The output shows the typical pattern. With 5 steps the left cluster is centered at −1.11-1.11 with standard deviation 0.390.39 — the samples are pulled toward the middle and smeared, because large steps follow a curved trajectory with straight lines. With 20 steps it is −1.37-1.37 and 0.300.30; with 100 steps −1.42-1.42 and 0.240.24, close to the data. No retraining was needed to change the step count.

Three scatter plots: an eight-component Gaussian mixture on a circle, samples from 1000-step DDPM, and samples from 10-step deterministic DDIM with trajectories drawn from noise to data.
Figure 3 — Samplers driven by the exact denoiser of an 8-component mixture, so only the sampling rule differs. DDPM with 1000 steps reproduces the data; deterministic DDIM with 10 steps lands near the modes but with slightly wider, smeared clusters. Black lines are DDIM trajectories from starting noise (×).

Faster solvers

Once sampling is “solve an ODE,” numerical analysis applies. Karras et al. (EDM) [20] used a second-order Heun solver with a carefully chosen spacing of noise levels and showed that sampler, schedule, preconditioning and training noise distribution can be tuned separately. DPM-Solver [21] exploits the semi-linear structure of the diffusion ODE — the linear part can be integrated exactly — to reach good quality in around 10 network evaluations. Stochastic samplers add a little fresh noise at each step; that costs more steps but can correct accumulated errors. Whatever the solver, report the number of function evaluations (NFE): the number of network calls, which, rather than “steps,” is the true sampling cost.

Steering generation: conditioning and guidance

Plain version. Conditioning specifies the requested output; guidance adjusts how strongly sampling favors it. More guidance need not mean a better image.

Conditioning

A conditional model learns ϵθ(xt,t,c)\epsilon_\theta(x_t,t,c) for a condition cc. Class labels enter through embeddings added to the time embedding; text enters through cross-attention from image features to text-encoder tokens [7]; images (for super-resolution, inpainting, editing) are concatenated to the input channels or fed through extra encoders.

Classifier guidance

Bayes’ rule for the noisy marginals gives ∇log⁡pt(xt∣y)=∇log⁡pt(xt)+∇log⁡pt(y∣xt)\nabla\log p_t(x_t\mid y)=\nabla\log p_t(x_t)+\nabla\log p_t(y\mid x_t). Dhariwal and Nichol [18] trained a separate classifier pϕ(y∣xt)p_\phi(y\mid x_t) on noisy images and added its gradient, scaled by ww, to an unconditional model. In noise units,

ϵ^=ϵθ(xt,t)−w 1−αˉt  ∇xtlog⁡pϕ(y∣xt).\hat\epsilon=\epsilon_\theta(x_t,t)-w\,\sqrt{1-\bar\alpha_t}\;\nabla_{x_t}\log p_\phi(y\mid x_t).

A scale w>1w\gt1 samples roughly from pt(xt) pt(y∣xt)wp_t(x_t)\,p_t(y\mid x_t)^w, a sharpened distribution that trades diversity for fidelity. Together with architecture improvements, this was the work that showed diffusion models beating GANs on ImageNet [35] sample quality [18]. Its drawback is the extra classifier, which must be trained on noisy inputs for every noise level.

Classifier-free guidance (CFG)

Ho and Salimans [6] removed the classifier. During training, the condition is replaced by a “null” token for a random fraction of examples, so one network learns both ϵθ(xt,t,c)\epsilon_\theta(x_t,t,c) and ϵθ(xt,t,∅)\epsilon_\theta(x_t,t,\varnothing). Bayes’ rule says the classifier gradient equals the difference of the conditional and unconditional scores, so the implicit-classifier version of the formula above is

ϵguided=ϵθ(xt,t,∅)+w(ϵθ(xt,t,c)−ϵθ(xt,t,∅)).\epsilon_{\mathrm{guided}}= \epsilon_\theta(x_t,t,\varnothing) +w\big(\epsilon_\theta(x_t,t,c)-\epsilon_\theta(x_t,t,\varnothing)\big).

In this convention w=0w=0 is unconditional, w=1w=1 is the ordinary conditional model, and w>1w\gt1 extrapolates past it. (Some papers write the same thing as (1+w′)ϵc−w′ϵu(1+w')\epsilon_c-w'\epsilon_u, so their “scale” is w−1w-1 in our notation; always check the convention before comparing numbers.)

The snippet below uses the exact noise predictor of a four-blob mixture, so any odd behaviour is due to guidance itself, not to a badly trained network. Class 1 is the two blobs with x>0x\gt0.

Code · Python · 32 lines
import numpy as np

# Data: four Gaussian blobs at the corners of a square; class c is the sign of x.
MU = np.array([[-1.5, -1.5], [-1.5, 1.5], [1.5, -1.5], [1.5, 1.5]])
SIG = 0.2
CLASS = np.array([0, 0, 1, 1])

def eps_exact(x, abar, comps):
    """Exact E[eps | x_t] for a Gaussian mixture restricted to `comps`."""
    m = np.sqrt(abar)*MU[comps]
    var = abar*SIG**2 + 1 - abar
    logw = -((x[:, None] - m[None])**2).sum(-1) / (2*var)
    w = np.exp(logw - logw.max(1, keepdims=True))
    w /= w.sum(1, keepdims=True)
    score = -(w[..., None]*(x[:, None] - m[None])).sum(1) / var
    return -np.sqrt(1-abar)*score

abar = np.cumprod(1 - np.linspace(1e-4, 0.02, 1000))
taus = np.linspace(999, 0, 50).round().astype(int)
everything, class1 = np.arange(4), np.where(CLASS == 1)[0]
print("data: mean distance to its mode", round(SIG*np.sqrt(np.pi/2), 3))
for w in [0, 1, 3, 8]:
    x = np.random.default_rng(0).normal(size=(4000, 2))
    for i, t in enumerate(taus):                    # deterministic DDIM
        e_u, e_c = eps_exact(x, abar[t], everything), eps_exact(x, abar[t], class1)
        e = e_u + w*(e_c - e_u)                     # classifier-free guidance
        x0 = (x - np.sqrt(1-abar[t])*e) / np.sqrt(abar[t])
        a_prev = abar[taus[i+1]] if i+1 < len(taus) else 1.0
        x = np.sqrt(a_prev)*x0 + np.sqrt(1-a_prev)*e
    dist = np.linalg.norm(x[:, None] - MU[None], axis=-1).min(1)
    print(f"w={w}: class-1 fraction {(x[:, 0] > 0).mean():.3f}, "
          f"mean distance to nearest mode {dist.mean():.3f}")

The data’s mean distance to its mode is 0.251. With w=0w=0 about half the samples (0.490) are class 1; with w=1w=1 all of them are, at distance 0.225 (50 DDIM steps slightly under-disperse). At w=3w=3 the distance is 0.259 and at w=8w=8 it grows to 0.446: strong guidance does not just “make it more class 1” — it pushes samples past the data, a toy version of the over-saturated colours and distorted shapes seen with very high guidance in image models. Figure 4 shows the same effect on the 8-mode mixture.

Four scatter plots of guided samples from an eight-mode mixture for guidance weights 0, 1, 3 and 8; higher weights stretch the class-1 clusters into elongated streaks.
Figure 4 — Exact classifier-free guidance on an 8-mode mixture whose class 1 has 4 modes (orange circles). w = 1 already restricts samples to class 1; larger w shrinks clusters along some directions and stretches them along others — the samples no longer follow any data distribution.

Guidance has a compute cost too: each step needs a conditional and an unconditional prediction, so a 30-step guided sampler costs 60 network evaluations unless the two are fused. And ww is a sampling knob, not a calibrated measure of how well the image matches the prompt.

Latent diffusion and diffusion transformers

Plain version. Pixels are expensive. Compress the image into a smaller grid of features with an autoencoder, run diffusion in that compressed space, then decode. Separately, the denoising network itself can be a transformer instead of a convolutional U-Net [37].

Latent diffusion

Rombach et al. [7] train an autoencoder first: an encoder z=E(x)z=\mathcal E(x) maps an H×W×3H\times W\times3 image to an h×w×ch\times w\times c latent with a spatial downsampling factor f=H/hf=H/h, and a decoder D\mathcal D maps back. The autoencoder is trained with reconstruction, perceptual and adversarial losses plus a light regularizer (a small KL term toward a Gaussian, or vector quantization) so that the latent space stays well behaved. Diffusion is then trained on zz exactly as before, and generation ends with x^=D(z0)\hat x=\mathcal D(z_0). With f=8f=8, a 512×512512\times512 image becomes a 64×6464\times64 latent: 64 times fewer spatial positions for every network call.

The trade-off is a reconstruction bottleneck: information discarded by the encoder cannot be faithfully recovered from that code, although the decoder may synthesize plausible detail. The latent’s scale must also match what the diffusion model was trained on; implementations rescale latents to roughly unit variance. Text conditioning in [7] uses cross-attention layers in the U-Net, the design behind the Stable Diffusion family.

Diffusion transformers

DiT (Peebles and Xie [8]) replaces the U-Net with a Vision Transformer [41] (see ML 05) over latent patches. A latent of size h×wh\times w is cut into p×pp\times p patches, giving hw/p2hw/p^2 tokens; smaller pp means more tokens, more compute and better quality. The timestep and class label condition every block through adaptive layer norm: they predict the scale and shift of each LayerNorm, and in the “adaLN-Zero” variant also a gating factor initialized to zero, so each residual block starts as the identity. The paper’s main finding is a scaling trend: across model sizes and patch sizes, more transformer compute (GFLOPs) gave consistently lower FID [8].

It helps to keep three choices separate. The architecture (U-Net or transformer) is the function approximator. The objective (noise, x0x_0, vv, or flow velocity) is what it regresses. The sampler (DDPM, DDIM, ODE solver) is how predictions become an image. Stable Diffusion 3 [9] is an example of a new combination: a latent transformer with separate weights for the image and text token streams that attend jointly (“MM-DiT”), trained with a rectified-flow objective and a timestep distribution biased toward intermediate noise levels.

For video, the latent and the attention must cover time as well as space, so compression and attention patterns become the main engineering problem. Melnik et al.’s survey [31] reviews architectures, temporal-consistency mechanisms and evaluation for video diffusion. Per-frame quality is not enough there: object permanence, motion plausibility and long-duration consistency need their own evaluation.

Pixel space, latent space and representation autoencoders

Plain version. The generator needs a place to do its repeated updates. That place can be the image itself, a compact reconstruction code, or features learned by a visual representation model. A latent stage buys a particular computational and statistical trade-off; it is not a prerequisite for generative modeling.

Why pixels become latents and then pixels again

Training images arrive as pixels, while the output must be viewable as pixels. Between these endpoints, an encoder can remove redundancy and produce a grid that is cheaper to update repeatedly [7]. Train the encoder and decoder on reconstruction first, then fit a generative distribution to encoded training images. These are two different tasks: reconstruction preserves a given example; generation learns which encoded examples are plausible.

The training and sampling routes are

training: x→Ezdata→corruptzt→prediction lossθ,generation: znoise→learned samplerz^→Dx^,reconstruction check: x→EE(x)→Dx~.\begin{aligned} \text{training: }&x\xrightarrow{\mathcal E}z_{\mathrm{data}} \xrightarrow{\text{corrupt}}z_t \xrightarrow{\text{prediction loss}}\theta,\\ \text{generation: }&z_{\mathrm{noise}}\xrightarrow{\text{learned sampler}}\hat z \xrightarrow{\mathcal D}\hat x,\\ \text{reconstruction check: }&x\xrightarrow{\mathcal E}\mathcal E(x) \xrightarrow{\mathcal D}\tilde x. \end{aligned}

Here xx is a training image; E\mathcal E and D\mathcal D are the encoder and decoder; zdataz_{\mathrm{data}} is an encoded image; ztz_t is its corrupted state; θ\theta parameterizes the generative predictor; z^\hat z is a generated code; x^\hat x is the generated image; and x~\tilde x is a reconstruction. Pure text-to-image sampling does not encode an input image first. Editing and image-to-image generation can encode an actual input image and begin from a partially corrupted code.

The main saving occurs because the denoiser or flow predictor runs repeatedly, while encoding training data and decoding a sample occur much less often. A spatial factor ff reduces grid positions by f2f^2, but the number of latent channels also matters. For an image with three channels and a latent with cc channels, the scalar-count reduction is 3f2/c3f^2/c. It is incorrect to describe the grid reduction as the reduction in all floating-point values or total runtime.

For a transformer, token count and token width must also be distinguished. Full attention stores a matrix with a number of entries proportional to N2N^2, where NN is the token count; feature projections depend on token width as well [41]. A patch size can reduce NN in either pixels or latents. None of these counts alone predicts wall-clock speed because kernels, memory traffic, the decoder and conditioning contribute.

Code · Python · 15 lines
# Synthetic shape bookkeeping, not a benchmark or a specific checkpoint.
H = W = 512
channels, spatial_factor, patch = 4, 8, 2
pixel_values = H*W*3
h = H//spatial_factor
w = W//spatial_factor
latent_values = h*w*channels
pixel_tokens = (H//patch)*(W//patch)
latent_tokens = (h//patch)*(w//patch)
print("Spatial positions reduction:", (H*W)//(h*w))
print("Scalar values reduction:", pixel_values//latent_values)
print("Dense attention entries reduction:", pixel_tokens**2//latent_tokens**2)
assert pixel_values//latent_values == 48
assert pixel_tokens//latent_tokens == 64
assert pixel_tokens**2//latent_tokens**2 == 4096

This constructed example yields 64 times fewer spatial positions, 48 times fewer scalar values and 4096 times fewer attention entries at the same numerical patch size in each coordinate space. Changing patch sizes changes the comparison. A pixel transformer with much larger patches can have the same token count as a latent transformer.

The cost is information loss. If two images have the same code, a deterministic decoder cannot recover their different exact details. A decoder may invent plausible texture, but that does not restore information about the original image. The reconstruction check therefore deserves its own evaluation before generative training. Inspect fine lines, lettering and temporal changes, rather than relying only on average image error.

The following deliberately simple encoder averages blocks and the decoder repeats those averages. It demonstrates a bottleneck without pretending to reproduce a learned VAE or RAE.

Code · Python · 13 lines
import numpy as np

y, x = np.mgrid[:64, :64]
checker = ((x+y) % 2).astype(float)
smooth = x/63
image = .6*smooth + .4*checker
f = 4
latent = image.reshape(64//f, f, 64//f, f).mean(axis=(1, 3))
reconstruction = np.repeat(np.repeat(latent, f, axis=0), f, axis=1)
print("Synthetic reconstruction MSE:", round(np.mean((image-reconstruction)**2), 6))
assert np.mean((image-reconstruction)**2) > .04
# Recovering the exact latent does not recover the missing checkerboard.
assert np.allclose(reconstruction[0, :f], reconstruction[0, 0])
Original synthetic image with a smooth ramp and checkerboard, its block-average reconstruction, and the absolute reconstruction error.
Figure 5 — A latent bottleneck can remove detail before the generative model is trained. This block-average toy is an explanation of information loss, not an evaluation of a learned autoencoder.

For editing, repeatedly encoding and decoding a result can accumulate distortion. Keep intermediate edits in the compatible latent space when the pipeline permits it, and inspect the eventual pixel output. A good unconditional generator may still be a poor reconstruction system. The claim that an autoencoder sets a ceiling concerns faithfulness to encoded information; it does not mean that a generative decoder can never synthesize new fine texture.

Reconstruction codes versus semantic representation codes

A VAE encoder is trained with a reconstruction objective and a distributional regularizer; its code is designed to support decoding and sampling [7]. A pretrained recognition encoder instead learns useful visual features. These goals can align, but recognition invariance may discard details needed for faithful reconstruction. The decoder and the retained feature structure determine whether the representation is a useful generative space.

Zheng, Ma, Tong and Xie’s Representation Autoencoders (RAEs) [43] pair a frozen pretrained representation encoder with a trained decoder. Their ICLR 2026 study uses semantic visual features as the space for generative modeling. It examines high-dimensional token features, dimension-dependent noise scheduling, decoder robustness and a shallow wide diffusion head. This changes the representation being generated, rather than simply adding a feature-alignment loss to a conventional latent predictor.

Do not equate “latent” with “smaller in every dimension.” A spatially sparse feature grid can have many channels per token. An RAE is also not automatically a variational model: a frozen feature extractor plus a decoder does not by itself define a Gaussian posterior, a KL regularizer or a likelihood bound. Keep reconstruction, representation quality and generative quality as separate measurements.

Tong et al. [44] extend this approach to freeform text-to-image generation in 2026. Their study compares RAE and VAE spaces at larger scales and examines decoder data composition. It finds that dimension-aware noise scheduling remains important while some small-scale architectural and decoder modifications offer little benefit at the tested larger scales. This is a reason to re-test recipes when scale and data change, rather than treating an ImageNet result as a universal design rule.

For a new application, inspect whether the chosen encoder preserves the content that matters: fine text for documents, small defects for inspection, or frame-to-frame differences for video. Evaluate the decoder on held-out real features and on generated features. A decoder trained only on exact training codes can be sensitive to small off-distribution errors in a sampled code. Normalize features using the training recipe, and record the encoder version with the generative checkpoint.

JiT: direct clean-image prediction on pixels

Li and He’s Just image Transformers (JiT) [42], published at CVPR 2026 after a 2025 preprint, revisit large-patch transformers operating directly on pixels. The network predicts the clean image; sampling derives a flow velocity from that prediction. Their controlled experiments support a manifold-based explanation: a bottleneck can retain structured clean-image information while making direct prediction of high-dimensional noise or velocity difficult. The reported setup uses no pretrained latent tokenizer or auxiliary representation loss. This is evidence for a particular architecture and training regime, not a proof that pixel-space models always outperform latent generators.

The crucial distinction is direct network output versus the quantity used in the loss or solver. For the straight path defined below, use yt=(1−t)ϵ+txy_t=(1-t)\epsilon+tx, with noise at t=0t=0 and clean data at t=1t=1. A network that directly predicts x^θ(yt,t)\hat x_\theta(y_t,t) can supply

u^θ(yt,t)=x^θ(yt,t)−yt1−t,∥u^θ−(x−ϵ)∥2=∥x^θ−x∥2(1−t)2.\hat u_\theta(y_t,t)=\frac{\hat x_\theta(y_t,t)-y_t}{1-t}, \qquad \|\hat u_\theta-(x-\epsilon)\|^2 =\frac{\|\hat x_\theta-x\|^2}{(1-t)^2}.

Here xx is the clean image; ϵ\epsilon is the noise draw; yty_t is their interpolation; x^θ\hat x_\theta is the direct prediction; and u^θ\hat u_\theta is the corresponding transport velocity. The identity holds for t<1t\lt1 before numerical clipping. An xx-output network can thus train with a velocity-space loss. Calling it “clean prediction” does not mean its loss is an unweighted clean-image MSE. JiT’s released implementation makes this distinction explicit [42].

Near t=1t=1, division by 1−t1-t needs an endpoint convention and numerical safeguards. For the interior algebra, a fixed clean-image error is amplified increasingly as the endpoint approaches. During sampling, use the checkpoint’s time distribution, noise scaling and solver recipe. Changing the direct output label alone does not reproduce the method.

Logarithmic plots of clean-image error amplification from a VP noise prediction and flow-velocity error amplification from a clean-image prediction.
Figure 6 — Equivalent predictions use different units. The left curve shows noise-error amplification at low signal; the right shows velocity-error amplification near the clean endpoint. The curves are algebraic identities, not model performance measurements.

A useful comparison fixes training data, compute, resolution, token count and the sampler, then changes the representation or target one at a time. Compare a pixel transformer, a compact VAE-latent transformer and a representation-latent transformer on reconstruction needs, generated quality and total cost. Record encoder/decoder cost and any representation pretraining. A model that uses a pretrained visual encoder has inherited training even when its generative stage starts from scratch.

Optional: JiT (in the Paper guides). Inspect the pixel architecture and the prediction-space ablation.

Flow matching: learn a transport velocity

Plain version. For a straight conditional path, pair noise with data and regress its velocity. Generation follows the learned velocity field, whose trajectories need not be straight.

Time convention change. In this section time runs from noise at t=0t=0 to data at t=1t=1, the opposite of the diffusion sections above (where large tt meant more noise).

The conditional flow-matching objective

A continuous normalizing flow moves points with an ODE dxt/dt=v(xt,t)dx_t/dt=v(x_t,t). If the velocity field vv is right, it transports the noise distribution p0=N(0,I)p_0=\mathcal N(0,I) to the data distribution p1p_1. Lipman et al. [10] (and, concurrently, Liu et al. [11] and Albergo and Vanden-Eijnden’s stochastic interpolants [34]) showed how to train vv without ever simulating the ODE.

Pair noise z∼N(0,I)z\sim\mathcal N(0,I) with a data example x∼p1x\sim p_1 and define the straight conditional path

xt=(1−t) z+t x,dxtdt=ut(xt∣z,x)=x−z.x_t=(1-t)\,z+t\,x,\qquad \frac{dx_t}{dt}=u_t(x_t\mid z,x)=x-z .

Every pair moves at constant speed in a straight line. The marginal velocity we actually want is the average of these conditional velocities over all pairs that pass through the point:

ut(y)=E[ x−z ∣ xt=y ].u_t(y)=\mathbb E\big[\,x-z\ \big|\ x_t=y\,\big].

We cannot compute this average, but we do not need to. Compare the “ideal” loss with the one we can compute:

LFM=Et,xt∥vθ(xt,t)−ut(xt)∥2,LCFM=Et,z,x∥vθ(xt,t)−(x−z)∥2,\mathcal L_{\mathrm{FM}}=\mathbb E_{t,x_t}\big\|v_\theta(x_t,t)-u_t(x_t)\big\|^2, \qquad \mathcal L_{\mathrm{CFM}}=\mathbb E_{t,z,x}\big\|v_\theta(x_t,t)-(x-z)\big\|^2,

with t∼U[0,1]t\sim\mathcal U[0,1]. Expand both squares. The ∥vθ∥2\|v_\theta\|^2 terms are identical. The cross terms are equal because, by the tower rule, E[vθ(xt,t)⊤(x−z)]=E[vθ(xt,t)⊤ E[x−z∣xt]]=E[vθ⊤ut(xt)]\mathbb E[v_\theta(x_t,t)^\top(x-z)]=\mathbb E\big[v_\theta(x_t,t)^\top\,\mathbb E[x-z\mid x_t]\big]=\mathbb E[v_\theta^\top u_t(x_t)]. The remaining terms do not involve θ\theta. Therefore

∇θ LCFM=∇θ LFM,\nabla_\theta\,\mathcal L_{\mathrm{CFM}}=\nabla_\theta\,\mathcal L_{\mathrm{FM}} ,

so regressing the per-pair velocity x−zx-z trains the marginal velocity field. This is the same trick as in diffusion: a target that is noisy for any single example has the right conditional mean. Generation solves dx/dt=vθ(x,t)dx/dt=v_\theta(x,t) from t=0t=0 to t=1t=1 with any ODE solver.

Random noise points connected by straight line segments to a set of data points on a circle.
Figure 7 — Conditional training paths are straight. Paths of different pairs cross, so the marginal velocity at a crossing point is an average, and the learned trajectories can curve.

How flow matching relates to diffusion

The linear path is a Gaussian path: given xx, xt∼N(t x, (1−t)2I)x_t\sim\mathcal N(t\,x,\ (1-t)^2I). Writing z^=E[z∣xt]\hat z=\mathbb E[z\mid x_t] (a noise prediction) and using x=(xt−(1−t)z)/tx=(x_t-(1-t)z)/t,

ut(xt)=xt−z^t,∇log⁡pt(xt)=−z^1−t,sout(xt)=xt+(1−t) ∇log⁡pt(xt)t.u_t(x_t)=\frac{x_t-\hat z}{t}, \qquad \nabla\log p_t(x_t)=-\frac{\hat z}{1-t}, \qquad\text{so}\qquad u_t(x_t)=\frac{x_t+(1-t)\,\nabla\log p_t(x_t)}{t}.

So a flow-matching velocity, a noise predictor and a score are again interchangeable for this path; what differs is the path (straight lines versus the VP curve), the weighting of noise levels in the loss, and the time convention. Diffusion’s probability-flow ODE is itself a flow along a particular Gaussian path. Lipman et al.’s guide [2] and the stochastic-interpolant framework [34] treat both as special cases of one construction. The practical advice is the same as for parameterizations: the targets are related by derivation, not interchangeable by renaming.

Code: flow matching on two clusters

This standalone example learns the same two-cluster distribution as the DDPM snippet with linear conditional paths, then samples with Euler steps of size 1/N1/N.

Code · Python · 32 lines
import torch
from torch import nn
from torch.nn import functional as F

torch.manual_seed(0)
torch.set_num_threads(1)
net = nn.Sequential(nn.Linear(3, 64), nn.SiLU(),
                    nn.Linear(64, 64), nn.SiLU(), nn.Linear(64, 2))
opt = torch.optim.Adam(net.parameters(), lr=.002)
for step in range(3000):
    z = torch.randn(128, 2)
    side = torch.randint(2, (128, 1))*2-1
    x = torch.cat([1.5*side, torch.zeros(128, 1)], 1) + .25*torch.randn(128, 2)
    t = torch.rand(128, 1)
    xt = (1-t)*z+t*x
    loss = F.mse_loss(net(torch.cat([xt, t], 1)), x-z)
    opt.zero_grad(); loss.backward(); opt.step()
net.eval()
def euler_sample(n, steps, seed=1):
    x = torch.randn(n, 2, generator=torch.Generator().manual_seed(seed))
    with torch.no_grad():
        for k in range(steps):
            t = torch.full((n, 1), k/steps)
            x = x + net(torch.cat([x, t], 1))/steps
    return x

for steps in [1, 4, 50]:
    s = euler_sample(2000, steps)
    assert torch.isfinite(s).all()
    left = s[s[:, 0] < 0]
    print(f"{steps:2d} Euler steps: left fraction {(s[:, 0] < 0).float().mean():.3f}, "
          f"left-cluster mean x {left[:, 0].mean():.2f}, std {left[:, 0].std():.2f}")

It prints, for 1, 4 and 50 steps, left-cluster means of −0.18-0.18, −1.20-1.20 and −1.45-1.45 with standard deviations 0.120.12, 0.420.42 and 0.290.29. One Euler step lands almost at the overall data mean: at t=0t=0 the marginal velocity is E[x]−z\mathbb E[x]-z, so a single full step sends every noise point to E[x]\mathbb E[x]. Straight conditional paths do not make the learned marginal flow straight. Figure 8 shows the same effect with the exact velocity of an 8-mode mixture.

Four scatter plots of Euler samples for 1, 2, 4 and 64 steps using the exact flow-matching velocity of an eight-mode mixture; one step collapses to the center, two steps form a disc, four steps a ring, 64 steps eight clusters with curved paths.
Figure 8 — Even with the exact marginal velocity, Euler sampling needs many steps: one step collapses every sample to the data mean, and the 64-step trajectories (black) bend toward the modes late in the path.

Rectified flow and reflow

Liu, Gong and Liu [11] use the same straight interpolation (“rectified flow”) and add reflow: after training, generate pairs (z,x^)(z,\hat x) by solving the learned ODE from each zz, then retrain on these coupled pairs instead of independent ones. Because the new pairs are already connected by the flow, their straight lines cross less, the learned trajectories become straighter, and fewer Euler steps are needed; a straight enough flow can then be distilled into a one-step model. Stable Diffusion 3 [9] showed that rectified-flow training of a large latent transformer works at text-to-image scale.

Optional: Flow Matching (in the Paper guides). Read the original figure, evidence, design choices and limitations.

After flow matching: intervals, image outputs and one-step pixels

Plain version. There are two separate questions after learning a flow: can one prediction cover a whole interval, and what should the network directly output? MeanFlow changes the interval being modeled; JiT changes the output representation. Pixel MeanFlow brings these directions together. Read them in that order instead of treating every new name as another sampler.

First align the clocks

The preceding flow-matching example moves from noise at zero to data at one. MeanFlow [13], improved MeanFlow [49] and pixel MeanFlow [50] use the reverse convention:

zt=(1−t)x+tϵ,t=0: data,t=1: noise.z_t=(1-t)x+t\epsilon,\qquad t=0:\text{ data},\quad t=1:\text{ noise}.

Here xx is a clean training example, ϵ\epsilon is prior noise, and ztz_t is their interpolated training state. Generation runs from one to zero. To compare with JiT’s time ss, use s=1−ts=1-t and reverse the velocity sign. This is a coordinate change, not a different generative principle. The conditional straight interpolation used to form training pairs still does not imply that the learned marginal ODE trajectories are straight.

MeanFlow: replace a tangent with an interval average

An instantaneous velocity tells you which way to move now. MeanFlow [13] learns the average over an interval, so a prediction can describe its entire displacement:

u(zt,r,t)=1t−r∫rtv(zs,s) ds,zr=zt−(t−r)u(zt,r,t).u(z_t,r,t)=\frac{1}{t-r}\int_r^t v(z_s,s)\,ds,\qquad z_r=z_t-(t-r)u(z_t,r,t).

Here vv is the marginal instantaneous velocity, zsz_s lies on its ODE trajectory, r<tr\lt t are interval endpoints, and uu is the average velocity along that trajectory. The zero-length boundary uses the limit u=vu=v. This integral defines the desired quantity; training does not obtain a teacher trajectory by evaluating it for every example.

Instead, differentiating the definition yields u=v−(t−r)du/dtu=v-(t-r)du/dt. The total derivative follows the moving state and holds rr fixed; it is not merely a partial derivative with respect to the time input. The original recipe uses a Jacobian-vector product and a stopped-gradient target constructed from the current network. One-call generation then uses the full interval (r,t)=(0,1)(r,t)=(0,1), provided the learned average is accurate enough.

The following exact curve illustrates the distinction. It is an algebra check, not a trained MeanFlow model:

Code · Python · 9 lines
r, t = 0.0, 1.0
start = (t, t*t)                    # curve z(t) = (t, t^2)
instant = (1.0, 2*t)                # tangent at the starting time
average = (1.0, t+r)                # exact interval-average derivative
one_euler = tuple(z-(t-r)*v for z, v in zip(start, instant))
one_average = tuple(z-(t-r)*u for z, u in zip(start, average))
assert one_average == (r, r*r)
print("Instantaneous Euler jump:", one_euler)
print("Average-velocity jump:", one_average)

The outputs are (0.0, -1.0) and (0.0, 0.0). A correct local tangent can miss the endpoint after one large step, whereas the correct interval average reaches it. Learning that average is the difficult part.

Optional: MeanFlow (in the Paper guides). Read the original figure, controlled comparisons and limitations.

Improved MeanFlow: make prediction inputs and guidance explicit

iMF [49] rewrites training as a loss on instantaneous velocity while the network still outputs average velocity. The important change is inside the derivative calculation: the original recipe feeds the sampled conditional velocity into the JVP; iMF uses a network prediction of marginal velocity. This keeps the clean training label out of the compound predictor’s inputs. It can reuse the zero-length boundary or use an auxiliary velocity head during training.

A separate improvement makes guidance scale and its active interval explicit conditions. A single trained model can then vary those settings at inference while retaining one-call generation. That is different from adding two conditional/unconditional network calls afterward. Its in-context condition tokens also change model size, so the final system comparison is not a pure loss-function ablation.

Optional: Improved MeanFlow (in the Paper guides). Read the original figure, controlled comparisons and limitations.

JiT: image output is a different design axis

The earlier JiT discussion separates direct clean-image output from velocity-space loss [42]. In the time convention used here, the derived instantaneous velocity is (zt−x^θ)/t(z_t-\widehat x_\theta)/t, where x^θ\widehat x_\theta is the predicted clean image. This conversion needs care near zero. JiT still follows a multi-step trajectory; an image-shaped output does not automatically provide an interval-average velocity or a one-step sample.

The controlled comparison is especially useful: hold loss space fixed, then change direct output and patch dimension. It explains why algebraic conversions between targets do not make finite-capacity networks equally easy to train. Do not compare only the strongest headline models with different training recipes.

Optional: JiT (in the Paper guides). Read the original figure, controlled comparisons and limitations.

Pixel MeanFlow: connect image output to the interval field

Pixel MeanFlow [50] defines a two-time image-like field x⋆(zt,r,t)=zt−t u(zt,r,t)x^\star(z_t,r,t)=z_t-t\,u(z_t,r,t). Here x⋆x^\star is induced by the average-velocity field, not necessarily the clean training image paired with ztz_t. The network predicts this field as xθx_\theta; converting back gives uθ=(zt−xθ)/tu_\theta=(z_t-x_\theta)/t. This conversion uses t>0t\gt0 with endpoint safeguards.

The derived uθu_\theta enters iMF’s instantaneous-velocity loss: image output → average velocity → instantaneous-velocity loss. For the full interval, the update simplifies to the network’s image output, enabling one-call pixel generation. At intermediate intervals, the image-like manifold interpretation is a hypothesis supported by the paper’s visualizations and ablations, not an exact manifold guarantee.

No latent tokenizer is needed at generation, but the stronger recipe uses pretrained feature networks for perceptual training losses. Record that training dependency when comparing it with JiT’s recipe or a latent generator. “No decoder,” “one NFE” and “no pretrained component anywhere” are three different claims.

Optional: Pixel MeanFlow (in the Paper guides). Read the original figure, controlled comparisons and limitations.

Keep neighboring approaches separate

Reflow [11] changes the coupling of training pairs to straighten trajectories. Consistency models [12] learn agreement of endpoint predictions along a trajectory; teacher-based distillation and teacher-free training are separate settings. MeanFlow instead parameterizes an interval average, while JiT addresses the network output space. The few-step overview compares these neighboring families.

Before choosing a method, write down its representation, direct output, loss target, time direction and sampling calls. Then compare matched resolution and conditioning, distinguish generator training from representation pretraining, and include decoder cost where relevant. A lower NFE is useful evidence about calls, not a complete latency or quality evaluation.

Pretrained image and video generators as complete pipelines

Plain version. A pretrained generator is a bundle of compatible components. Loading the denoiser weights alone does not establish the right representation, text conditioning, prediction target or sampling schedule. The image/video boundary also changes what a reconstruction must preserve.

Read the components before running the checkpoint

A practical pipeline combines a text encoder, an optional image/video encoder, a time-conditioned predictor, a sampler and an optional decoder. A separate refinement model may follow the first generator. Each component expects particular tensor shapes and scales. A scheduler declaring v_prediction may mean VP velocity; a flow scheduler may expect a derivative along a straight interpolation. Check the training path rather than guessing from a configuration string.

The following verified examples illustrate architectural differences. They are fixed research baselines, rather than a ranking of the latest product releases. The references include their papers or technical reports and official release documentation.

BaselineGeneration representation and predictorWhat to inspect in the pretrained pipeline
SDXL [45]Reconstruction latents with a convolutional denoiserText conditioning, latent scaling, optional base/refiner composition and compatible sampling configuration
Stable Diffusion 3 [9]Latent rectified flow with a joint text–image transformerFlow time convention, text streams, time sampling and autoencoder compatibility
FLUX.1 [46]Latent rectified-flow transformerExact release variant, guidance behavior, text conditioning, autoencoder and release terms
JiT [42]Class-conditioned clean-image prediction directly on pixelsImage normalization, patch/resolution configuration, derived velocity and sampling recipe; this is not a freeform text-to-image checkpoint
HunyuanVideo [47]Compressed video latents with a flow-matching transformerVideo autoencoder, image/video joint training, temporal dimensions and text-conditioning components
Wan2.1 [48]Video diffusion transformer with a spatiotemporal autoencoderText-to-video versus image-to-video variant, causal video compression, temporal length and decode caching

SDXL [45] documents a larger latent image-generation architecture and optional refinement stage. Stable Diffusion 3 [9] illustrates how the same broad latent pipeline can instead use rectified flow and joint text/image attention. FLUX.1’s official model card [46] describes a released rectified-flow transformer; its .1-dev variant is guidance-distilled. These facts do not make the checkpoints interchangeable. Ordinary two-pass CFG and a distilled guidance input can have different meanings and costs.

For image-to-image generation, encoding an input and adding partial noise trades input preservation against the ability to change it. The actual noise level is determined by the schedule, not merely by a UI label such as “strength.” To compare edits, keep the same input and record the initial time, seed, prompt, guidance and reconstruction baseline. A reconstruction artifact can otherwise be mistaken for an edit requested by the text.

Video adds temporal compression and a consistency problem

Let a video have FF frames, height HH and width WW. A hypothetical encoder compressing time by ftf_t and each spatial axis by fsf_s gives a rough latent grid of (F/ft)×(H/fs)×(W/fs)(F/f_t)\times(H/f_s)\times(W/f_s) positions. Exact shapes depend on padding and special handling of the first frame. After patching, dense attention across all positions grows quadratically with the remaining token count; separately reducing time and space can therefore change cost substantially.

HunyuanVideo’s technical report [47] combines a 3D video autoencoder with a flow-matching transformer and staged image/video training. Wan’s report [48] explains a causal video VAE with special first-frame handling, spatial and temporal compression, and feature caching during chunked encoding/decoding. Causality in an autoencoder does not by itself imply that the entire video generator samples autoregressively; the generative attention and sampling procedure must be examined separately.

A frame can look excellent while a clip changes an object’s identity, reverses motion unexpectedly or forgets an occluded object. Test reconstruction of held-out clips before generation, then evaluate appearance, motion, prompt correspondence and consistency across frames. Melnik et al. [31] provide the survey map for these video-specific issues. A per-image metric alone cannot diagnose temporal behavior.

A pretrained video generator also does not automatically become a reliable world model. Predicting plausible clips differs from predicting the consequences of a specified action. The world-model chapter develops this distinction through state, actions, rollouts and evaluation. Use the actual conditioning interface and evaluate the task the system claims to solve.

A reproducible first experiment

Start by recording a pipeline manifest: checkpoint revision, encoder/decoder revisions, text encoder, representation shape and scaling, prediction target, time direction, solver, evaluation count, guidance, dtype and seed. Then perform a reconstruction check if an autoencoder is present and generate a fixed small prompt/seed set. Compare one change at a time. Include decoder latency and peak memory when reporting the cost of a latent method.

This chapter’s CPU examples test the underlying identities and failure modes without downloading weights. Full-resolution pretrained generation belongs in a separately provisioned experiment using the official release instructions [42], [45]–[48]. Treat an actual measured run, a paper’s published result and a toy calculation as three different kinds of evidence.

Few-step and one-step generation

Plain version. If a sampler needs 50 network calls, train a second model that jumps along the trajectory in a few calls — or train one from scratch that is designed to jump.

The main families are:

  • Progressive distillation [22]: a student learns to match two deterministic teacher steps with one step; repeating the procedure halves the step count each round.
  • Consistency models [12]: learn a function fθ(xt,t)f_\theta(x_t,t) that maps any point on a probability-flow ODE trajectory to that trajectory’s starting (clean) point. It must satisfy the boundary condition fθ(x,ϵ)=xf_\theta(x,\epsilon)=x at the smallest time ϵ\epsilon, enforced by writing fθ(x,t)=cskip(t) x+cout(t) Fθ(x,t)f_\theta(x,t)=c_{\mathrm{skip}}(t)\,x+c_{\mathrm{out}}(t)\,F_\theta(x,t) with cskip(ϵ)=1c_{\mathrm{skip}}(\epsilon)=1 and cout(ϵ)=0c_{\mathrm{out}}(\epsilon)=0. Training asks outputs at adjacent points of a trajectory to agree. In consistency distillation, adjacent points come from one ODE-solver step of a pretrained diffusion model; in consistency training, from the same data example noised at two levels, with no teacher. Sampling takes one step, or a few steps of “jump to clean, re-noise, jump again.” Improved training recipes [32] simplified continuous-time consistency models and scaled them to 1.5 billion parameters on ImageNet 512×512512\times512 with two-step sampling.
  • Adversarial distillation: ADD [33] combines score distillation from a large pretrained diffusion teacher with an adversarial loss so that a student produces sharp images in one to four steps.
  • Average-velocity models: MeanFlow [13] learns interval displacement through an average velocity; iMF [49] improves its regression and guidance, and pixel MeanFlow [50] connects it to image output. The post-flow walkthrough develops these differences and their separate pretraining dependencies.

A small step count is a learned capability, not a free consequence of a nicer equation. When comparing, count NFE including guidance evaluations, the decoder’s cost in latent models, resolution, quality and diversity — and remember that a one-step network can be larger and slower per call than a multi-step one.

Evaluating generative models

Plain version. “Does it look good?” hides at least three questions: are individual samples realistic (fidelity), does the set of samples cover the variety of the data (diversity), and does the sample match the condition (alignment)? No single number answers all three.

Fréchet Inception Distance

FID (Heusel et al. [23]) embeds real and generated images with an Inception-v3 network [36], fits a Gaussian to each set of features, and computes the Fréchet (2-Wasserstein) distance between the two Gaussians:

FID=∥μr−μg∥2+tr⁡ ⁣(Σr+Σg−2(ΣrΣg)1/2),\mathrm{FID}=\|\mu_r-\mu_g\|^2+\operatorname{tr}\!\Big(\Sigma_r+\Sigma_g-2\big(\Sigma_r\Sigma_g\big)^{1/2}\Big),

where μr,Σr\mu_r,\Sigma_r and μg,Σg\mu_g,\Sigma_g are the feature mean and covariance of the real and generated sets and tr⁡\operatorname{tr} is the matrix trace. Lower is better. The snippet applies the formula to two-dimensional “features.”

Code · Python · 21 lines
import numpy as np
from scipy import linalg

def frechet_distance(a, b):
    """Frechet distance between Gaussians fitted to feature sets a and b."""
    mu1, mu2 = a.mean(0), b.mean(0)
    c1, c2 = np.cov(a, rowvar=False), np.cov(b, rowvar=False)
    covmean = linalg.sqrtm(c1 @ c2).real
    return float(((mu1 - mu2)**2).sum() + np.trace(c1 + c2 - 2*covmean))

rng = np.random.default_rng(0)
centers = np.array([[-1.5, 0.], [1.5, 0.]])
real = centers[rng.integers(2, size=5000)] + .25*rng.normal(size=(5000, 2))
good = centers[rng.integers(2, size=5000)] + .25*rng.normal(size=(5000, 2))
one_mode = centers[[0]] + .25*rng.normal(size=(5000, 2))      # mode collapse
blurry = .6*rng.normal(size=(5000, 2))                         # mass in the gap
print("good     :", round(frechet_distance(real, good), 3))
print("one mode :", round(frechet_distance(real, one_mode), 3))
print("blurry   :", round(frechet_distance(real, blurry), 3))
small = [frechet_distance(real[:50], good[i*50:(i+1)*50]) for i in range(20)]
print("good, 50 samples: mean", round(np.mean(small), 3), "std", round(np.std(small), 3))

It prints about 0.001 for a correct sampler, 3.9 for a sampler that only produces the left mode, and 0.976 for a “blurry” sampler that puts its mass between the modes. The last line shows a practical trap: with 50 samples, even a perfect sampler scores 0.077±0.0730.077\pm0.073, because FID is biased upward for small sample sizes. Compare FIDs only with the same number of samples, the same reference set and the same feature extractor and preprocessing.

Beyond FID

  • Precision and recall (Kynkäänniemi et al. [24]) separate fidelity from diversity: estimate the real and generated “manifolds” with kk-nearest-neighbour balls in feature space; precision is the fraction of generated samples inside the real manifold, and recall the fraction of real samples inside the generated one. Our “one mode” sampler would have high precision and low recall.
  • Feature extractor matters. Stein et al. [25] ran a large human study and found that no common metric correlated strongly with human judgments; in particular, the perceptual realism of diffusion models was not reflected in FID. They recommend features from DINOv2-ViT-L/14 [39] instead of Inception, and show that current metrics cannot separate memorization from underfitting or mode shrinkage.
  • CMMD (Jayasumana et al. [26]) replaces Inception features with CLIP [38] embeddings and the Gaussian assumption with a maximum mean discrepancy using a Gaussian RBF kernel, giving an unbiased estimator that is more sample-efficient.
  • Likelihood. A probability-flow ODE supports log-density evaluation through change of variables [4]; numerical tolerances and divergence estimation must be reported. A DDPM variational bound [19] is a different quantity. Converting a continuous density to image bpd also requires the stated dequantization and scaling convention. Neither score alone measures visual quality.
  • Conditional models need alignment checks (does the image show what the prompt says?), usually by human rating or by vision–language models, and memorization checks against the training set.
  • Cost. Report NFE, guidance evaluations, resolution and wall-clock time on stated hardware.

The evaluation chapter ML 11 returns to sample-size effects and honest comparisons.

Modern view

Surveys and reference texts. The literature is large; these reviews are good maps.

  • Yang et al. [1] (ACM Computing Surveys, 2023) organize diffusion research into three improvement directions — efficient sampling, improved likelihood estimation, and data with special structure (manifolds, discrete data, invariances) — and also cover combinations with VAEs, GANs and flows and applications across domains.
  • Croitoru et al. [27] (IEEE TPAMI, 2023) focus on vision. They identify three generic formulations — DDPMs, noise-conditioned score networks and SDEs — and categorize vision applications such as generation, super-resolution, inpainting, editing, translation and segmentation.
  • Cao et al. [28] (IEEE TKDE, 2024) take a methods angle: fundamental formulation, algorithmic enhancements (sampling acceleration, objective design, conditioning) and applications from images to biology and healthcare.
  • Lipman et al. [2] (2024) is a guide-with-code to flow matching that covers Gaussian and general probability paths, the link to diffusion and extensions to discrete and Riemannian data.
  • Lai, Song, Kim, Mitsufuji and Ermon [29] (2025) is a book-length monograph presenting the variational, score-based and flow-based views as three routes to the same time-dependent velocity field, followed by guidance, fast solvers and flow-map (few-step) models.
  • Shen et al. [30] (TMLR, 2025) survey efficiency at the algorithm level (training, sampling, compression), system level (hardware and serving) and framework level.
  • Melnik et al. [31] (TMLR, 2024) survey video diffusion: architectures, temporal-consistency mechanisms, datasets and evaluation.

State of the art, 2023–2026. Several trends are visible across these reviews and the primary papers in this chapter. (1) Transformers in latent space: DiT-style backbones [8] replaced U-Nets in many large systems, and joint text–image transformers trained with rectified-flow objectives [9] became a standard text-to-image recipe. (2) Flow matching as the default objective: straight-path flow matching [10], [11] is now used alongside or instead of noise prediction, with diffusion understood as one choice of probability path [2], [29]. (3) Fewer evaluations: distillation [22], [33], consistency models [12], [32] and average-velocity models [13] bring sampling down to one to four network calls with quality approaching multi-step samplers. (4) Better evaluation: the community is moving from Inception-based FID toward richer feature spaces, human studies and alignment metrics [25], [26]. (5) Beyond images: video, 3-D, audio and scientific data reuse the same machinery, with video’s temporal consistency and cost the main open challenges [31].

Representation and prediction, 2025–2026. JiT [42] tests direct clean-image prediction on pixels; RAE [43] tests generation in frozen semantic feature spaces; its text-to-image follow-up [44] examines larger-scale behavior. These studies separate where the model operates from what the network directly predicts. The pretrained examples above [45]–[48] show why component compatibility and temporal reconstruction matter in actual image/video pipelines. Compare these choices under matched compute and data, including representation pretraining.

Open problems. Sampling cost at high resolution and long video duration; principled guidance that does not trade away diversity or produce over-saturation; evaluation metrics that agree with people and detect memorization; understanding generalization (why models produce new samples rather than copies of training data); controllability and editing with guarantees; and data provenance, consent and misuse, which are social rather than purely technical questions. The next chapter, ML 09, covers the main alternative route: autoregressive generation over pixels and tokens, and its hybrids with diffusion.

Key takeaways

  • Known forward corruption supplies training targets at sampled noise levels.
  • The simplified denoising objective changes the weighting of variational-bound terms; it is not itself the exact likelihood.
  • Prediction targets convert under a specified path, but weighting, endpoint behavior and model capacity still affect training.
  • A sampler needs a compatible schedule and prediction convention; report network evaluations and full latency.
  • Guidance changes the sampled distribution; stronger conditioning can reduce diversity and introduce artifacts.
  • Straight conditional training paths do not guarantee straight learned trajectories. Compare representations, solvers and few-step training under matched evaluation protocols.

Exercises

  1. Recover the clean estimate. Starting from xt=αˉtx0+1−αˉtϵx_t=\sqrt{\bar\alpha_t}x_0+\sqrt{1-\bar\alpha_t}\epsilon, derive x^0\hat x_0 from a noise prediction. If ϵθ\epsilon_\theta has an error δ\delta, how large is the error in x^0\hat x_0 at αˉt=0.25\bar\alpha_t=0.25 and at αˉt=10−4\bar\alpha_t=10^{-4}? Then derive the straight-flow velocity from a clean prediction and explain why JiT can output xx while using a velocity loss.
Hint

x^0=(xt−1−αˉt ϵθ)/αˉt\hat x_0=(x_t-\sqrt{1-\bar\alpha_t}\,\epsilon_\theta)/\sqrt{\bar\alpha_t}, so the error is 1−αˉt δ/αˉt\sqrt{1-\bar\alpha_t}\,\delta/\sqrt{\bar\alpha_t}: about 1.7 δ1.7\,\delta at αˉt=0.25\bar\alpha_t=0.25 and about 100 δ100\,\delta at 10−410^{-4}. At high noise, x^0\hat x_0 is an average over many plausible images, not one image.

  1. Posterior mean in two forms. Substitute x0=(xt−1−αˉt ϵ)/αˉtx_0=(x_t-\sqrt{1-\bar\alpha_t}\,\epsilon)/\sqrt{\bar\alpha_t} into μ~t\tilde\mu_t and show that it equals 1αt(xt−βt1−αˉtϵ)\frac{1}{\sqrt{\alpha_t}}\big(x_t-\frac{\beta_t}{\sqrt{1-\bar\alpha_t}}\epsilon\big). Then change t in the posterior snippet to 50 and 900 and check that the formula still matches.
Hint

Use αˉt=αtαˉt−1\bar\alpha_t=\alpha_t\bar\alpha_{t-1} and βt=1−αt\beta_t=1-\alpha_t. The coefficient of xtx_t becomes βt+αt(1−αˉt−1)αt(1−αˉt)=1αt\frac{\beta_t+\alpha_t(1-\bar\alpha_{t-1})}{\sqrt{\alpha_t}(1-\bar\alpha_t)}=\frac{1}{\sqrt{\alpha_t}} because the numerator equals 1−αˉt1-\bar\alpha_t.

  1. Count compute. A latent text-to-image model samples with 30 DDIM steps and classifier-free guidance using separate conditional and unconditional passes, then runs the decoder once. How many denoiser evaluations are needed? How does the count change for a 4-step distilled model that has guidance “baked in”? For the synthetic shape example, change the latent channels to 16 and the pixel patch size to 16. Recompute scalar counts and attention entries. Which savings remain, and which depend on patching?
Hint

60 denoiser evaluations plus one decoder pass; batching both CFG branches reduces wall-clock time but not the work. A guidance-distilled 4-step model needs 4 evaluations plus the decoder. Compare quality and diversity, not just the counts.

  1. Time convention. Diffusion sections use “large tt = noisy”; the flow-matching section uses ”t=0t=0 noise, t=1t=1 data.” Reparameterize the flow with s=1−ts=1-t. What is dx/dsdx/ds, and what goes wrong if you only relabel the time input?
Hint

dx/ds=−v(x,1−s)dx/ds=-v(x,1-s). If you feed ss to a network trained on tt without flipping the sign and the time input, you integrate in the wrong direction and move samples away from the data.

  1. Guidance sweep. In the CFG snippet, add w∈{0.5, 1.5, 5}w\in\{0.5,\,1.5,\,5\} and also record the fraction of class-1 samples that land closer to the wrong-class modes. Plot mean distance to the nearest mode against ww. Where does guidance stop helping?
Hint

Already at w=1w=1 the exact conditional model puts every sample in class 1, so larger ww cannot improve class accuracy here — it only distorts the clusters. With a learned, imperfect conditional model, moderate w>1w\gt1 can help adherence, which is why it is used in practice.

  1. Sampler ablation. Hold the trained flow-matching network fixed. Compare Euler with 1, 2, 5, 20 and 100 steps against a second-order Heun step (predict with Euler, then average the velocities at both ends). Use the same starting noise and measure cluster means and spreads. Does the 2-step Heun sampler beat 4-step Euler at the same number of network calls?
Hint

Heun uses two network calls per step. Plot samples, not only numbers. One Euler step of a model trained for instantaneous velocity does not test a separately trained one-step model such as a consistency or MeanFlow model.

Paper guides

6 paper guides, each on a page of its own.

Open the paper guides →

References

  1. L. Yang, Z. Zhang, Y. Song, S. Hong, R. Xu, Y. Zhao, W. Zhang, B. Cui and M.-H. Yang, “Diffusion Models: A Comprehensive Survey of Methods and Applications,” ACM Computing Surveys, 2023. arXiv
  2. Y. Lipman, M. Havasi, P. Holderrieth, N. Shaul, M. Le, B. Karrer, R. T. Q. Chen, D. Lopez-Paz, H. Ben-Hamu and I. Gat, “Flow Matching Guide and Code,” arXiv:2412.06264, 2024. arXiv
  3. J. Ho, A. Jain and P. Abbeel, “Denoising Diffusion Probabilistic Models,” NeurIPS, 2020. arXiv
  4. Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon and B. Poole, “Score-Based Generative Modeling through Stochastic Differential Equations,” ICLR, 2021. arXiv
  5. J. Song, C. Meng and S. Ermon, “Denoising Diffusion Implicit Models,” ICLR, 2021. arXiv
  6. J. Ho and T. Salimans, “Classifier-Free Diffusion Guidance,” NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications; arXiv:2207.12598, 2022. arXiv
  7. R. Rombach, A. Blattmann, D. Lorenz, P. Esser and B. Ommer, “High-Resolution Image Synthesis with Latent Diffusion Models,” CVPR, 2022. arXiv
  8. W. Peebles and S. Xie, “Scalable Diffusion Models with Transformers,” ICCV, 2023. arXiv
  9. P. Esser, S. Kulal, A. Blattmann et al., “Scaling Rectified Flow Transformers for High-Resolution Image Synthesis,” ICML, PMLR 235, 2024. PMLR
  10. Y. Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel and M. Le, “Flow Matching for Generative Modeling,” ICLR, 2023. arXiv
  11. X. Liu, C. Gong and Q. Liu, “Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow,” ICLR, 2023. arXiv
  12. Y. Song, P. Dhariwal, M. Chen and I. Sutskever, “Consistency Models,” ICML, 2023. arXiv
  13. Z. Geng, M. Deng, X. Bai, J. Z. Kolter and K. He, “Mean Flows for One-step Generative Modeling,” NeurIPS, 2025. arXiv
  14. S. J. D. Prince, Understanding Deep Learning, MIT Press, 2023. book site
  15. J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan and S. Ganguli, “Deep Unsupervised Learning using Nonequilibrium Thermodynamics,” ICML, PMLR 37, 2015. PMLR
  16. P. Vincent, “A Connection Between Score Matching and Denoising Autoencoders,” Neural Computation, vol. 23, no. 7, pp. 1661–1674, 2011. doi
  17. Y. Song and S. Ermon, “Generative Modeling by Estimating Gradients of the Data Distribution,” NeurIPS, 2019. arXiv
  18. P. Dhariwal and A. Nichol, “Diffusion Models Beat GANs on Image Synthesis,” NeurIPS, 2021. NeurIPS
  19. A. Q. Nichol and P. Dhariwal, “Improved Denoising Diffusion Probabilistic Models,” ICML, PMLR 139, 2021. PMLR
  20. T. Karras, M. Aittala, T. Aila and S. Laine, “Elucidating the Design Space of Diffusion-Based Generative Models,” NeurIPS, 2022. arXiv
  21. C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li and J. Zhu, “DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps,” NeurIPS, 2022. arXiv
  22. T. Salimans and J. Ho, “Progressive Distillation for Fast Sampling of Diffusion Models,” ICLR, 2022. arXiv
  23. M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler and S. Hochreiter, “GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium,” NeurIPS (NIPS), 2017. arXiv
  24. T. Kynkäänniemi, T. Karras, S. Laine, J. Lehtinen and T. Aila, “Improved Precision and Recall Metric for Assessing Generative Models,” NeurIPS, 2019. arXiv
  25. G. Stein, J. C. Cresswell, R. Hosseinzadeh et al., “Exposing Flaws of Generative Model Evaluation Metrics and Their Unfair Treatment of Diffusion Models,” NeurIPS, 2023. arXiv
  26. S. Jayasumana, S. Ramalingam, A. Veit, D. Glasner, A. Chakrabarti and S. Kumar, “Rethinking FID: Towards a Better Evaluation Metric for Image Generation,” CVPR, 2024. CVF
  27. F.-A. Croitoru, V. Hondru, R. T. Ionescu and M. Shah, “Diffusion Models in Vision: A Survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023. doi
  28. H. Cao, C. Tan, Z. Gao, Y. Xu, G. Chen, P.-A. Heng and S. Z. Li, “A Survey on Generative Diffusion Models,” IEEE Transactions on Knowledge and Data Engineering, vol. 36, no. 7, pp. 2814–2830, 2024. doi
  29. C.-H. Lai, Y. Song, D. Kim, Y. Mitsufuji and S. Ermon, “The Principles of Diffusion Models,” monograph, arXiv:2510.21890, 2025. arXiv
  30. H. Shen, J. Zhang, B. Xiong et al., “Efficient Diffusion Models: A Survey,” Transactions on Machine Learning Research, 2025. arXiv
  31. A. Melnik, M. Ljubljanac, C. Lu, Q. Yan, W. Ren and H. Ritter, “Video Diffusion Models: A Survey,” Transactions on Machine Learning Research, 2024. arXiv
  32. C. Lu and Y. Song, “Simplifying, Stabilizing and Scaling Continuous-Time Consistency Models,” ICLR, 2025. arXiv
  33. A. Sauer, D. Lorenz, A. Blattmann and R. Rombach, “Adversarial Diffusion Distillation,” ECCV, 2024. arXiv
  34. M. S. Albergo and E. Vanden-Eijnden, “Building Normalizing Flows with Stochastic Interpolants,” ICLR, 2023. arXiv
  35. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li and L. Fei-Fei, “ImageNet: A Large-Scale Hierarchical Image Database,” CVPR, 2009, pp. 248–255. doi
  36. C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens and Z. Wojna, “Rethinking the Inception Architecture for Computer Vision,” CVPR, 2016, pp. 2818–2826. arXiv
  37. O. Ronneberger, P. Fischer and T. Brox, “U-Net: Convolutional Networks for Biomedical Image Segmentation,” MICCAI, LNCS 9351, pp. 234–241, 2015. arXiv
  38. A. Radford, J. W. Kim, C. Hallacy et al., “Learning Transferable Visual Models From Natural Language Supervision,” ICML, 2021. arXiv
  39. M. Oquab, T. Darcet, T. Moutakanni et al., “DINOv2: Learning Robust Visual Features without Supervision,” Transactions on Machine Learning Research, 2024. arXiv
  40. B. Efron, “Tweedie’s Formula and Selection Bias,” Journal of the American Statistical Association, vol. 106, no. 496, pp. 1602–1614, 2011. doi
  41. A. Dosovitskiy, L. Beyer, A. Kolesnikov et al., “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale,” ICLR, 2021. arXiv
  42. T. Li and K. He, “Back to Basics: Let Denoising Generative Models Denoise,” CVPR, 2026 (preprint 2025). paper, publication record, official JiT code.
  43. B. Zheng, N. Ma, S. Tong and S. Xie, “Diffusion Transformers with Representation Autoencoders,” ICLR, 2026 (preprint 2025). proceedings, manuscript.
  44. S. Tong, B. Zheng, Z. Wang, B. Tang, N. Ma, E. Brown, J. Yang, R. Fergus, Y. LeCun and S. Xie, “Scaling Text-to-Image Diffusion Transformers with Representation Autoencoders,” CVPR Findings Track, 2026. manuscript, author publication record.
  45. D. Podell, Z. English, K. Lacey, A. Blattmann, T. Dockhorn, J. Müller, J. Penna and R. Rombach, “SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis,” ICLR, 2024. proceedings, official model card.
  46. Black Forest Labs, “FLUX.1 [dev],” official pretrained-model card, 2024. model card.
  47. W. Kong, Q. Tian, Z. Zhang et al., “HunyuanVideo: A Systematic Framework For Large Video Generative Models,” technical report, arXiv:2412.03603, 2024. report, official release.
  48. Team Wan, A. Wang, B. Ai, B. Wen et al., “Wan: Open and Advanced Large-Scale Video Generative Models,” technical report, arXiv:2503.20314, 2025. report, official Wan2.1 release.
  49. Z. Geng, Y. Lu, Z. Wu, E. Shechtman, J. Z. Kolter and K. He, “Improved Mean Flows: On the Challenges of Fastforward Generative Models,” CVPR, pp. 30467–30476, 2026. CVF
  50. Y. Lu, S. Lu, Q. Sun, H. Zhao, Z. Jiang, X. Wang, T. Li, Z. Geng and K. He, “One-step Latent-free Image Generation with Pixel Mean Flows,” ICML, PMLR 306, pp. 81265–81277, 2026. PMLR

Check your understanding

This tutorial has 6 exercises. Can you solve them without opening the hints?

Go to the exercises ↑

Next ML 09 · Autoregressive Vision: Pixels, Tokens and Scales · 28 min