Stretch space, change density: the Jacobian matters
BeginnerML 01B · Probability and Information TheoryInteractive plot
Description
Push a Gaussian through an invertible nonlinear map and preserve probability mass with its derivative.
Try this: Increase the nonlinear stretch near zero. Why does the output density at zero fall even though zero still maps to zero?
Hints
The chapter section that uses this lab:
Lab · Stretch space, change density: the Jacobian matters
- Derivative at zero
- 2
- Output density at zero
- 0.199
- Numerical area (±4 base σ)
- 0.99988
pₓ(f(z)) = p_z(z) / |f′(z)|. A local stretch spreads the same probability over more output space. Omitting the Jacobian would change the total probability mass.
Try this: Increase the nonlinear stretch near zero. Why does the output density at zero fall even though zero still maps to zero?
One-dimensional normalizing flow x=az+b tanh(z), with a>0 and b≥0. The derivative is always positive, so the map is invertible. Curves are computed from the exact change-of-variables formula, without fitting a model.
Method sources
- Ian Goodfellow, Yoshua Bengio & Aaron Courville, Deep Learning, MIT Press, 2016
- Ashish Vaswani et al., Attention Is All You Need, NeurIPS, 2017
- Jonathan Ho, Ajay Jain & Pieter Abbeel, Denoising Diffusion Probabilistic Models, NeurIPS, 2020
- Tim Salimans & Jonathan Ho, Progressive Distillation for Fast Sampling of Diffusion Models, ICLR, 2022