ML 08 · What does a network see when it predicts noise, the image or a velocity?
AdvancedML 08 · Diffusion Models and Flow MatchingImage lab
Description
Mix the astronaut photograph with fixed noise, look at the noise, clean-image and velocity targets side by side, then give the network a small error and see which conversion back to the clean image survives.
Try this: Start at −20 dB with an error of 0.05 in the noise output. Slide the signal level up to 0 dB: where do the three recoveries become equally good? Then go to +20 dB: which target is now the most error-sensitive, and why does the answer flip?
Hints
The chapter section that uses this lab:
Explanation
Answer the prediction first, or open the explanation when you are ready.
Show the explanation
The conversions are exact, but they do not treat errors alike. The noise route divides by √ᾱ, so an error in ε̂ is multiplied by √(1−ᾱ)/√ᾱ, which is 10.0 at −20 dB; the clean-image route passes the error through unchanged (×1), and the velocity route multiplies it by √(1−ᾱ), which is 0.995. With the same error of 0.05 the recovered image has a PSNR of 13.5 dB from the noise route, against 32.4 dB from the clean-image route and 32.4 dB from the velocity route. Ironically the noise is the target that is easiest to read off the picture at this level: ε is almost a copy of x_t (correlation 0.998), while v is nearly the negative of the clean image (correlation −0.988). And the ranking flips near the clean end: at +20 dB the clean-image route is the most sensitive (32.4 dB, against 52.3 dB for the other two).
Lab · ML 08 · What does a network see when it predicts noise, the image or a velocity?
Loading image…
Try this: Start at −20 dB with an error of 0.05 in the noise output. Slide the signal level up to 0 dB: where do the three recoveries become equally good? Then go to +20 dB: which target is now the most error-sensitive, and why does the answer flip?
Image: the astronaut photograph (256 × 256, RGB), with pixel values mapped to [-1, 1] as in the chapter. The mix is x_t = √ᾱ x₀ + √(1−ᾱ) ε with a fixed seeded standard-normal ε, and v = √ᾱ ε − √(1−ᾱ) x₀ is the velocity of the ML 08 chapter (the diffusion velocity, not the flow-matching transport velocity). The slider sets the signal-to-noise ratio ᾱ/(1−ᾱ) of x_t in dB. The network is not trained: its output is the exact target plus a fixed seeded error, and the same error size (in the units of the [-1, 1] pixel scale) is applied whichever target is predicted. A real network does not make equal errors for every target, so this isolates the conversion, not the learning. x̂₀ is clipped to [-1, 1] as an image decoder would, and PSNR is measured against x₀ with both mapped to [0, 1] (peak 1). All panels share one gray mapping (gray = 0.5 + u/2.5).
Method and image sources
- Jonathan Ho, Ajay Jain & Pieter Abbeel, Denoising Diffusion Probabilistic Models, NeurIPS 2020 (forward mix x_t and noise prediction)
- Tim Salimans & Jonathan Ho, Progressive Distillation for Fast Sampling of Diffusion Models, ICLR 2022 (the velocity target v)
- scikit-image test images (skimage.data): camera is CC0 by Lav Varshney; coins is from the Brooklyn Museum collection, no known copyright restrictions; astronaut is a NASA photograph in the public domain