Mixed Precision Training — where precision matters
Optional · paper guide 1 of 1 for ML 01C · Numerical Computation and Gradient Checks. Citation numbers refer to the reference list at the bottom of this page.
Navigate: this page and this track: Navigate: this page and this track
Question and thesis
Can training use cheaper arithmetic without losing the small changes that make learning work? Micikevicius et al. propose a coordinated recipe, not simply converting every tensor to half precision [12]. Read floating-point range, rounding and gradient descent first. The central distinction is between storing an activation, accumulating a sum and updating a parameter: these operations need not use the same format.
Method and figure

Follow Figure 1 around the update loop. An FP32 master weight is copied into FP16 for layer computation. Products can use reduced precision while accumulation and sensitive operations retain FP32. The optimizer updates the master copy, so an update too small to change an FP16 weight immediately can still accumulate across steps.
Loss scaling addresses a different problem: gradients can fall below the representable FP16 range before reaching the optimizer. Write
Here is the original loss, is a positive scale held fixed during this differentiation, is the scaled loss, denotes parameters and is the unscaled gradient passed to the update. This preserves the intended gradient in exact arithmetic while changing intermediate magnitudes. Unscaling must occur before an operation whose result depends on the gradient magnitude.
Evidence
The paper tests image classification, detection and speech systems rather than only one network. Figure 2 shows why maintaining master weights matters in its speech experiment. Figure 3 examines the small-gradient distribution in its detector; a loss scale of 8 suffices in the reported SSD experiment. That is an observed setting for that model, not a universal safe constant. Read the task-specific accuracy comparisons separately from claims about hardware throughput.
Design analysis
Master weights protect small parameter updates; scaling protects small gradients. Neither replaces the other. Scaling too aggressively can instead cause overflow. A fair comparison keeps the model, data and training schedule fixed and checks accuracy as well as time and memory. The FP32 master copy also means that total training memory is not automatically halved, even when many activations use FP16.
Reading checkpoint
A weight rounds to the same FP16 value after one update. Explain how repeated FP32 master updates can eventually change its FP16 copy. Then identify why that mechanism cannot recover a gradient already rounded to zero during backpropagation. These are two distinct failure locations in the figure.
Limits and next step
The paper studies its particular FP16/FP32 recipe and supported hardware. Do not transfer its range analysis unchanged to another floating-point format or promise the same speedup for a memory-bound workload. Continue with ML 02 optimization: record the numeric format, scaling policy and optimizer state precision alongside the learning rate when diagnosing training.
Continue reading: Adam — adaptive updates and the convergence caveat
References
- I. Goodfellow, Y. Bengio and A. Courville, Deep Learning, MIT Press, 2016, ch. 4 (Numerical Computation) and ch. 8. Book
- D. Goldberg, “What Every Computer Scientist Should Know About Floating-Point Arithmetic,” ACM Computing Surveys, vol. 23, no. 1, pp. 5–48, 1991. DOI
- IEEE, “IEEE Standard for Floating-Point Arithmetic,” IEEE Std 754-2019, 2019. DOI
- N. J. Higham, Accuracy and Stability of Numerical Algorithms, 2nd ed., SIAM, 2002. DOI
- D. Kalamkar et al., “A Study of BFLOAT16 for Deep Learning Training,” arXiv:1905.12322, 2019. arXiv
- J. Nocedal and S. J. Wright, Numerical Optimization, 2nd ed., Springer, 2006. DOI
- S. Boyd and L. Vandenberghe, Convex Optimization, Cambridge University Press, 2004. Book
- Y. N. Dauphin, R. Pascanu, C. Gulcehre, K. Cho, S. Ganguli and Y. Bengio, “Identifying and Attacking the Saddle Point Problem in High-Dimensional Non-Convex Optimization,” NeurIPS, 2014. Paper
- B. A. Pearlmutter, “Fast Exact Multiplication by the Hessian,” Neural Computation, vol. 6, no. 1, pp. 147–160, 1994. DOI
- D. C. Liu and J. Nocedal, “On the Limited Memory BFGS Method for Large Scale Optimization,” Mathematical Programming, vol. 45, pp. 503–528, 1989. DOI
- A. G. Baydin, B. A. Pearlmutter, A. A. Radul and J. M. Siskind, “Automatic Differentiation in Machine Learning: a Survey,” Journal of Machine Learning Research, vol. 18, no. 153, pp. 1–43, 2018. arXiv
- P. Micikevicius et al., “Mixed Precision Training,” ICLR, 2018. arXiv
- PyTorch documentation, “Numerical Accuracy.” Documentation
- L. Bottou, F. E. Curtis and J. Nocedal, “Optimization Methods for Large-Scale Machine Learning,” SIAM Review, vol. 60, no. 2, pp. 223–311, 2018. DOI
- J. M. Cohen, S. Kaur, Y. Li, J. Z. Kolter and A. Talwalkar, “Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability,” ICLR, 2021. arXiv
- J. Martens and R. Grosse, “Optimizing Neural Networks with Kronecker-factored Approximate Curvature,” ICML, 2015. Paper
- V. Gupta, T. Koren and Y. Singer, “Shampoo: Preconditioned Stochastic Tensor Optimization,” ICML, 2018. Paper
- J. Liu, J. Su, X. Yao et al., “Muon is Scalable for LLM Training,” arXiv:2502.16982, 2025. arXiv
- P. Micikevicius et al., “FP8 Formats for Deep Learning,” arXiv:2209.05433, 2022. arXiv
- M. Milakov and N. Gimelshein, “Online Normalizer Calculation for Softmax,” arXiv:1805.02867, 2018. arXiv
- T. Dao, D. Y. Fu, S. Ermon, A. Rudra and C. Ré, “FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness,” NeurIPS, 2022. Paper
- J. Shah, G. Bikshandi, Y. Zhang, V. Thakkar, P. Ramani and T. Dao, “FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision,” NeurIPS, 2024. Paper
- NumPy developers, “numpy.linalg.cond,” NumPy documentation, accessed 2026-11-05. Documentation
- PyTorch contributors, “torch.autograd.gradcheck.gradcheck,” PyTorch documentation, accessed 2026-11-05. Documentation