PR 08 · Which direction of KL are you minimizing?
AdvancedPR 08 · Approximate Inference and Bayesian Model ComplexityInteractive plot
Description
Move and widen a Gaussian approximation to a two-mode target; compare penalties for missing mass and filling the valley.
Try this: With separated modes, compare a narrow approximation on one mode against a wide one covering both. Which KL prefers each?
Hints
The chapter section that uses this lab:
Explanation
Answer the prediction first, or open the explanation when you are ready.
Show the explanation
Reverse KL punishes q for putting mass where p has none, so a narrow q settles on one bump (mode seeking). Forward KL punishes q for missing any of p's mass, so its best q is wide and covers both bumps. In this setup the best forward fit has mean 0 and σ about 2.1; the best reverse fit sits on a bump at ±2 with σ about 0.7.
Lab · PR 08 · Which direction of KL are you minimizing?
- KL(p ‖ q)
- 8.777
- KL(q ‖ p)
- 0.69
KL(p‖q) weights mismatch where the target has mass; KL(q‖p) weights mismatch where the approximation puts mass. A single Gaussian may cover both modes or settle on one, depending on the objective.
Try this: With separated modes, compare a narrow approximation on one mode against a wide one covering both. Which KL prefers each?
Target p is an equal mixture N(−a,0.65²) and N(a,0.65²). Approximation q is N(μ,σ²). Both are numerically normalized on [−12,12] with spacing 0.02; KL values are grid approximations in nats. This is an adjustable distribution family, not an optimizer.