Full search page →

Mixed Precision Training — where precision matters

Optional · paper guide 1 of 1 for ML 01C · Numerical Computation and Gradient Checks. Citation numbers refer to the reference list at the bottom of this page.

Navigate: this page and this track: Navigate: this page and this track

Question and thesis

Can training use cheaper arithmetic without losing the small changes that make learning work? Micikevicius et al. propose a coordinated recipe, not simply converting every tensor to half precision [12]. Read floating-point range, rounding and gradient descent first. The central distinction is between storing an activation, accumulating a sum and updating a parameter: these operations need not use the same format.

Method and figure

The training loop keeps an FP32 master copy for updates while using FP16 copies for much of the layer computation. Storage precision and accumulation precision have different roles.
Source: Micikevicius et al., ICLR 2018 manuscript, arXiv:1710.03740v3. Figure 1, PDF p. 3. The training loop keeps an FP32 master copy for updates while using FP16 copies for much of the layer computation. Storage precision and accumulation precision have different roles. [12]

Follow Figure 1 around the update loop. An FP32 master weight is copied into FP16 for layer computation. Products can use reduced precision while accumulation and sensitive operations retain FP32. The optimizer updates the master copy, so an update too small to change an FP16 weight immediately can still accumulate across steps.

Loss scaling addresses a different problem: gradients can fall below the representable FP16 range before reaching the optimizer. Write

Ls=SL,g=1S∇wLs.L_s=S L,\qquad g=\frac{1}{S}\nabla_w L_s.

Here LL is the original loss, SS is a positive scale held fixed during this differentiation, LsL_s is the scaled loss, ww denotes parameters and gg is the unscaled gradient passed to the update. This preserves the intended gradient in exact arithmetic while changing intermediate magnitudes. Unscaling must occur before an operation whose result depends on the gradient magnitude.

Evidence

The paper tests image classification, detection and speech systems rather than only one network. Figure 2 shows why maintaining master weights matters in its speech experiment. Figure 3 examines the small-gradient distribution in its detector; a loss scale of 8 suffices in the reported SSD experiment. That is an observed setting for that model, not a universal safe constant. Read the task-specific accuracy comparisons separately from claims about hardware throughput.

Design analysis

Master weights protect small parameter updates; scaling protects small gradients. Neither replaces the other. Scaling too aggressively can instead cause overflow. A fair comparison keeps the model, data and training schedule fixed and checks accuracy as well as time and memory. The FP32 master copy also means that total training memory is not automatically halved, even when many activations use FP16.

Reading checkpoint

A weight rounds to the same FP16 value after one update. Explain how repeated FP32 master updates can eventually change its FP16 copy. Then identify why that mechanism cannot recover a gradient already rounded to zero during backpropagation. These are two distinct failure locations in the figure.

Limits and next step

The paper studies its particular FP16/FP32 recipe and supported hardware. Do not transfer its range analysis unchanged to another floating-point format or promise the same speedup for a memory-bound workload. Continue with ML 02 optimization: record the numeric format, scaling policy and optimizer state precision alongside the learning rate when diagnosing training.

Continue reading: Adam — adaptive updates and the convergence caveat

References

  1. I. Goodfellow, Y. Bengio and A. Courville, Deep Learning, MIT Press, 2016, ch. 4 (Numerical Computation) and ch. 8. Book
  2. D. Goldberg, “What Every Computer Scientist Should Know About Floating-Point Arithmetic,” ACM Computing Surveys, vol. 23, no. 1, pp. 5–48, 1991. DOI
  3. IEEE, “IEEE Standard for Floating-Point Arithmetic,” IEEE Std 754-2019, 2019. DOI
  4. N. J. Higham, Accuracy and Stability of Numerical Algorithms, 2nd ed., SIAM, 2002. DOI
  5. D. Kalamkar et al., “A Study of BFLOAT16 for Deep Learning Training,” arXiv:1905.12322, 2019. arXiv
  6. J. Nocedal and S. J. Wright, Numerical Optimization, 2nd ed., Springer, 2006. DOI
  7. S. Boyd and L. Vandenberghe, Convex Optimization, Cambridge University Press, 2004. Book
  8. Y. N. Dauphin, R. Pascanu, C. Gulcehre, K. Cho, S. Ganguli and Y. Bengio, “Identifying and Attacking the Saddle Point Problem in High-Dimensional Non-Convex Optimization,” NeurIPS, 2014. Paper
  9. B. A. Pearlmutter, “Fast Exact Multiplication by the Hessian,” Neural Computation, vol. 6, no. 1, pp. 147–160, 1994. DOI
  10. D. C. Liu and J. Nocedal, “On the Limited Memory BFGS Method for Large Scale Optimization,” Mathematical Programming, vol. 45, pp. 503–528, 1989. DOI
  11. A. G. Baydin, B. A. Pearlmutter, A. A. Radul and J. M. Siskind, “Automatic Differentiation in Machine Learning: a Survey,” Journal of Machine Learning Research, vol. 18, no. 153, pp. 1–43, 2018. arXiv
  12. P. Micikevicius et al., “Mixed Precision Training,” ICLR, 2018. arXiv
  13. PyTorch documentation, “Numerical Accuracy.” Documentation
  14. L. Bottou, F. E. Curtis and J. Nocedal, “Optimization Methods for Large-Scale Machine Learning,” SIAM Review, vol. 60, no. 2, pp. 223–311, 2018. DOI
  15. J. M. Cohen, S. Kaur, Y. Li, J. Z. Kolter and A. Talwalkar, “Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability,” ICLR, 2021. arXiv
  16. J. Martens and R. Grosse, “Optimizing Neural Networks with Kronecker-factored Approximate Curvature,” ICML, 2015. Paper
  17. V. Gupta, T. Koren and Y. Singer, “Shampoo: Preconditioned Stochastic Tensor Optimization,” ICML, 2018. Paper
  18. J. Liu, J. Su, X. Yao et al., “Muon is Scalable for LLM Training,” arXiv:2502.16982, 2025. arXiv
  19. P. Micikevicius et al., “FP8 Formats for Deep Learning,” arXiv:2209.05433, 2022. arXiv
  20. M. Milakov and N. Gimelshein, “Online Normalizer Calculation for Softmax,” arXiv:1805.02867, 2018. arXiv
  21. T. Dao, D. Y. Fu, S. Ermon, A. Rudra and C. Ré, “FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness,” NeurIPS, 2022. Paper
  22. J. Shah, G. Bikshandi, Y. Zhang, V. Thakkar, P. Ramani and T. Dao, “FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision,” NeurIPS, 2024. Paper
  23. NumPy developers, “numpy.linalg.cond,” NumPy documentation, accessed 2026-11-05. Documentation
  24. PyTorch contributors, “torch.autograd.gradcheck.gradcheck,” PyTorch documentation, accessed 2026-11-05. Documentation