Full search page →

PR 08 · Which direction of KL are you minimizing?

AdvancedPR 08 · Approximate Inference and Bayesian Model ComplexityInteractive plot

Description

Move and widen a Gaussian approximation to a two-mode target; compare penalties for missing mass and filling the valley.

Try this: With separated modes, compare a narrow approximation on one mode against a wide one covering both. Which KL prefers each?

Predict first

The target has two separated bumps and q is a single Gaussian. Which divergence is smallest for a q that sits on just one bump?

Hints

The chapter section that uses this lab:

Explanation

Answer the prediction first, or open the explanation when you are ready.

Show the explanation

Reverse KL punishes q for putting mass where p has none, so a narrow q settles on one bump (mode seeking). Forward KL punishes q for missing any of p's mass, so its best q is wide and covers both bumps. In this setup the best forward fit has mean 0 and σ about 2.1; the best reverse fit sits on a bump at ±2 with σ about 0.7.

Lab · PR 08 · Which direction of KL are you minimizing?

Predict first

The target has two separated bumps and q is a single Gaussian. Which divergence is smallest for a q that sits on just one bump?

KL(p ‖ q)
8.777
KL(q ‖ p)
0.69

KL(p‖q) weights mismatch where the target has mass; KL(q‖p) weights mismatch where the approximation puts mass. A single Gaussian may cover both modes or settle on one, depending on the objective.

Try this: With separated modes, compare a narrow approximation on one mode against a wide one covering both. Which KL prefers each?

Target p is an equal mixture N(−a,0.65²) and N(a,0.65²). Approximation q is N(μ,σ²). Both are numerically normalized on [−12,12] with spacing 0.02; KL values are grid approximations in nats. This is an adjustable distribution family, not an optimizer.

Method sources

Open in the Playground →