Full search page →

Follow a gradient down a narrow valley

AdvancedOPT 06 · Gradient Descent, Newton Steps and Numerical ReliabilityInteractive plot

Description

See why a step size that works on a round bowl can diverge on a narrow one.

Try this: Set k = 10. Compare step sizes 0.1, 0.2 and 0.3. What happens to the y coordinate at the stability boundary?

Predict first

The curvature ratio is k = 5. Which step size makes the iterate blow up instead of settling?

Explanation

Answer the prediction first, or open the explanation when you are ready.

Show the explanation

The steep direction is multiplied by 1 − ηk each step, which has magnitude below 1 only when η is below 2/k = 0.4. So 0.15 and 0.30 converge (0.30 overshoots back and forth) and 0.45 diverges.

Lab · Follow a gradient down a narrow valley

Predict first

The curvature ratio is k = 5. Which step size makes the iterate blow up instead of settling?

Final objective
0.034
Converges when 0 < η <
0.4
Coordinate multipliers (x, y)
0.85, 0.25

Both coordinate multipliers have magnitude below 1, so the iterates converge. A negative multiplier produces an alternating path; a multiplier close to 1 or −1 produces slow progress.

Try this: Set k = 10. Compare step sizes 0.1, 0.2 and 0.3. What happens to the y coordinate at the stability boundary?

An exact quadratic: f(x,y) = (x² + ky²)/2, starting at (3,2). Updates are x ← (1−η)x and y ← (1−ηk)y. The contour plot has a fixed viewing window; displayed objective values are never clipped.

Method sources

Open in the Playground →