Follow a gradient down a narrow valley
AdvancedOPT 06 · Gradient Descent, Newton Steps and Numerical ReliabilityInteractive plot
Description
See why a step size that works on a round bowl can diverge on a narrow one.
Try this: Set k = 10. Compare step sizes 0.1, 0.2 and 0.3. What happens to the y coordinate at the stability boundary?
Hints
The chapter section that uses this lab:
Explanation
Answer the prediction first, or open the explanation when you are ready.
Show the explanation
The steep direction is multiplied by 1 − ηk each step, which has magnitude below 1 only when η is below 2/k = 0.4. So 0.15 and 0.30 converge (0.30 overshoots back and forth) and 0.45 diverges.
Lab · Follow a gradient down a narrow valley
- Final objective
- 0.034
- Converges when 0 < η <
- 0.4
- Coordinate multipliers (x, y)
- 0.85, 0.25
Both coordinate multipliers have magnitude below 1, so the iterates converge. A negative multiplier produces an alternating path; a multiplier close to 1 or −1 produces slow progress.
Try this: Set k = 10. Compare step sizes 0.1, 0.2 and 0.3. What happens to the y coordinate at the stability boundary?
An exact quadratic: f(x,y) = (x² + ky²)/2, starting at (3,2). Updates are x ← (1−η)x and y ← (1−ηk)y. The contour plot has a fixed viewing window; displayed objective values are never clipped.