完整搜尋頁面 →

論文導讀

選讀:ML 01C・數值計算與梯度檢查 所依據論文的導讀,每篇一頁:說明它要解決的問題、核心想法、結果,以及閱讀時該注意什麼。

導覽:本頁與本系列: 導覽:本頁與本系列

以下為選讀導讀。先讀問題與原圖,再逐層展開證據、設計分析及閱讀檢查點。

  1. Mixed Precision Training:精度該保留在哪裡

    訓練能否使用較便宜的算術,又不丟失讓學習成立的微小變化?

參考文獻

  1. I. Goodfellow, Y. Bengio and A. Courville, Deep Learning, MIT Press, 2016, ch. 4 (Numerical Computation) and ch. 8. Book
  2. D. Goldberg, “What Every Computer Scientist Should Know About Floating-Point Arithmetic,” ACM Computing Surveys, vol. 23, no. 1, pp. 5–48, 1991. DOI
  3. IEEE, “IEEE Standard for Floating-Point Arithmetic,” IEEE Std 754-2019, 2019. DOI
  4. N. J. Higham, Accuracy and Stability of Numerical Algorithms, 2nd ed., SIAM, 2002. DOI
  5. D. Kalamkar et al., “A Study of BFLOAT16 for Deep Learning Training,” arXiv:1905.12322, 2019. arXiv
  6. J. Nocedal and S. J. Wright, Numerical Optimization, 2nd ed., Springer, 2006. DOI
  7. S. Boyd and L. Vandenberghe, Convex Optimization, Cambridge University Press, 2004. Book
  8. Y. N. Dauphin, R. Pascanu, C. Gulcehre, K. Cho, S. Ganguli and Y. Bengio, “Identifying and Attacking the Saddle Point Problem in High-Dimensional Non-Convex Optimization,” NeurIPS, 2014. Paper
  9. B. A. Pearlmutter, “Fast Exact Multiplication by the Hessian,” Neural Computation, vol. 6, no. 1, pp. 147–160, 1994. DOI
  10. D. C. Liu and J. Nocedal, “On the Limited Memory BFGS Method for Large Scale Optimization,” Mathematical Programming, vol. 45, pp. 503–528, 1989. DOI
  11. A. G. Baydin, B. A. Pearlmutter, A. A. Radul and J. M. Siskind, “Automatic Differentiation in Machine Learning: a Survey,” Journal of Machine Learning Research, vol. 18, no. 153, pp. 1–43, 2018. arXiv
  12. P. Micikevicius et al., “Mixed Precision Training,” ICLR, 2018. arXiv
  13. PyTorch documentation, “Numerical Accuracy.” Documentation
  14. L. Bottou, F. E. Curtis and J. Nocedal, “Optimization Methods for Large-Scale Machine Learning,” SIAM Review, vol. 60, no. 2, pp. 223–311, 2018. DOI
  15. J. M. Cohen, S. Kaur, Y. Li, J. Z. Kolter and A. Talwalkar, “Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability,” ICLR, 2021. arXiv
  16. J. Martens and R. Grosse, “Optimizing Neural Networks with Kronecker-factored Approximate Curvature,” ICML, 2015. Paper
  17. V. Gupta, T. Koren and Y. Singer, “Shampoo: Preconditioned Stochastic Tensor Optimization,” ICML, 2018. Paper
  18. J. Liu, J. Su, X. Yao et al., “Muon is Scalable for LLM Training,” arXiv:2502.16982, 2025. arXiv
  19. P. Micikevicius et al., “FP8 Formats for Deep Learning,” arXiv:2209.05433, 2022. arXiv
  20. M. Milakov and N. Gimelshein, “Online Normalizer Calculation for Softmax,” arXiv:1805.02867, 2018. arXiv
  21. T. Dao, D. Y. Fu, S. Ermon, A. Rudra and C. Ré, “FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness,” NeurIPS, 2022. Paper
  22. J. Shah, G. Bikshandi, Y. Zhang, V. Thakkar, P. Ramani and T. Dao, “FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision,” NeurIPS, 2024. Paper
  23. NumPy developers, “numpy.linalg.cond,” NumPy documentation, accessed 2026-11-05. Documentation
  24. PyTorch contributors, “torch.autograd.gradcheck.gradcheck,” PyTorch documentation, accessed 2026-11-05. Documentation