Why a smaller finite-difference step can be worse
BeginnerML 01C · Numerical Computation and Gradient ChecksInteractive plot
Description
Check the derivative of exp(x), separating truncation error from floating-point cancellation.
Try this: Sweep h from 10⁻¹ to 10⁻¹⁶. Find the useful range for forward and central differences rather than choosing the smallest step.
Hints
The chapter section that uses this lab:
Lab · Why a smaller finite-difference step can be worse
- Analytic derivative
- 2.71828183
- Numerical derivative
- 2.71828183
- Relative error
- 2.155e-11
Large h introduces approximation error. Very small h subtracts nearly identical floating-point values, losing useful digits. Central differences cancel the leading truncation term but still suffer roundoff.
Try this: Sweep h from 10⁻¹ to 10⁻¹⁶. Find the useful range for forward and central differences rather than choosing the smallest step.
Computed with JavaScript binary64 arithmetic; exact means the analytic derivative exp(x) evaluated in the same arithmetic. A zero measured error is rounded agreement, not proof of exact real arithmetic.
Method sources
- Ian Goodfellow, Yoshua Bengio & Aaron Courville, Deep Learning, MIT Press, 2016
- Ashish Vaswani et al., Attention Is All You Need, NeurIPS, 2017
- Jonathan Ho, Ajay Jain & Pieter Abbeel, Denoising Diffusion Probabilistic Models, NeurIPS, 2020
- Tim Salimans & Jonathan Ho, Progressive Distillation for Fast Sampling of Diffusion Models, ICLR, 2022