Full search page →

Why a smaller finite-difference step can be worse

BeginnerML 01C · Numerical Computation and Gradient ChecksInteractive plot

Description

Check the derivative of exp(x), separating truncation error from floating-point cancellation.

Try this: Sweep h from 10⁻¹ to 10⁻¹⁶. Find the useful range for forward and central differences rather than choosing the smallest step.

Lab · Why a smaller finite-difference step can be worse

Analytic derivative
2.71828183
Numerical derivative
2.71828183
Relative error
2.155e-11

Large h introduces approximation error. Very small h subtracts nearly identical floating-point values, losing useful digits. Central differences cancel the leading truncation term but still suffer roundoff.

Try this: Sweep h from 10⁻¹ to 10⁻¹⁶. Find the useful range for forward and central differences rather than choosing the smallest step.

Computed with JavaScript binary64 arithmetic; exact means the analytic derivative exp(x) evaluated in the same arithmetic. A zero measured error is rounded agreement, not proof of exact real arithmetic.

Method sources

Open in the Playground →