Full search page →

ML 08 · What does a network see when it predicts noise, the image or a velocity?

AdvancedML 08 · Diffusion Models and Flow MatchingImage lab

Description

Mix the astronaut photograph with fixed noise, look at the noise, clean-image and velocity targets side by side, then give the network a small error and see which conversion back to the clean image survives.

Try this: Start at −20 dB with an error of 0.05 in the noise output. Slide the signal level up to 0 dB: where do the three recoveries become equally good? Then go to +20 dB: which target is now the most error-sensitive, and why does the answer flip?

Predict first

The picture x_t is almost pure noise (−20 dB). A network makes the same small error, 0.05, whichever target it predicts. After converting each prediction back to a clean image x̂₀, which one is the most error-sensitive?

Hints

The chapter section that uses this lab:

Explanation

Answer the prediction first, or open the explanation when you are ready.

Show the explanation

The conversions are exact, but they do not treat errors alike. The noise route divides by √ᾱ, so an error in ε̂ is multiplied by √(1−ᾱ)/√ᾱ, which is 10.0 at −20 dB; the clean-image route passes the error through unchanged (×1), and the velocity route multiplies it by √(1−ᾱ), which is 0.995. With the same error of 0.05 the recovered image has a PSNR of 13.5 dB from the noise route, against 32.4 dB from the clean-image route and 32.4 dB from the velocity route. Ironically the noise is the target that is easiest to read off the picture at this level: ε is almost a copy of x_t (correlation 0.998), while v is nearly the negative of the clean image (correlation −0.988). And the ranking flips near the clean end: at +20 dB the clean-image route is the most sensitive (32.4 dB, against 52.3 dB for the other two).

Lab · ML 08 · What does a network see when it predicts noise, the image or a velocity?

Predict first

The picture x_t is almost pure noise (−20 dB). A network makes the same small error, 0.05, whichever target it predicts. After converting each prediction back to a clean image x̂₀, which one is the most error-sensitive?

Loading image…

x_t, the network input: the only picture it gets to see
ε, the noise that was added
x₀, the clean image
v = √ᾱ·ε − √(1−ᾱ)·x₀, the velocity
x̂₀ recovered from the network’s imperfect output of the chosen target
Error x̂₀ − x₀ (gain ×3, mid-gray = 0)

Try this: Start at −20 dB with an error of 0.05 in the noise output. Slide the signal level up to 0 dB: where do the three recoveries become equally good? Then go to +20 dB: which target is now the most error-sensitive, and why does the answer flip?

Image: the astronaut photograph (256 × 256, RGB), with pixel values mapped to [-1, 1] as in the chapter. The mix is x_t = √ᾱ x₀ + √(1−ᾱ) ε with a fixed seeded standard-normal ε, and v = √ᾱ ε − √(1−ᾱ) x₀ is the velocity of the ML 08 chapter (the diffusion velocity, not the flow-matching transport velocity). The slider sets the signal-to-noise ratio ᾱ/(1−ᾱ) of x_t in dB. The network is not trained: its output is the exact target plus a fixed seeded error, and the same error size (in the units of the [-1, 1] pixel scale) is applied whichever target is predicted. A real network does not make equal errors for every target, so this isolates the conversion, not the learning. x̂₀ is clipped to [-1, 1] as an image decoder would, and PSNR is measured against x₀ with both mapped to [0, 1] (peak 1). All panels share one gray mapping (gray = 0.5 + u/2.5).

Method and image sources

Open in the Playground →