論文導讀
選讀:ML 01B・機率與資訊理論 所依據論文的導讀,每篇一頁:說明它要解決的問題、核心想法、結果,以及閱讀時該注意什麼。
導覽:本頁與本系列: 導覽:本頁與本系列
以下為選讀導讀。先讀問題與原圖,再逐層展開證據、設計分析及閱讀檢查點。
- Shannon:通訊的數學理論
來源能產生多少資訊,通道又能多可靠地傳遞?
參考文獻
- I. Goodfellow, Y. Bengio and A. Courville, Deep Learning, MIT Press, 2016, ch. 3 “Probability and Information Theory”. Book
- C. M. Bishop and H. Bishop, Deep Learning: Foundations and Concepts, Springer, 2024. Book
- K. P. Murphy, Probabilistic Machine Learning: An Introduction, MIT Press, 2022. Book
- M. P. Deisenroth, A. A. Faisal and C. S. Ong, Mathematics for Machine Learning, Cambridge University Press, 2020. Book
- C. E. Shannon, “A Mathematical Theory of Communication,” Bell System Technical Journal, vol. 27, pp. 379–423, 1948. DOI
- S. Kullback and R. A. Leibler, “On Information and Sufficiency,” The Annals of Mathematical Statistics, vol. 22, pp. 79–86, 1951. DOI
- T. M. Cover and J. A. Thomas, Elements of Information Theory, 2nd ed., Wiley, 2006. DOI
- D. P. Kingma and M. Welling, “Auto-Encoding Variational Bayes,” in Proc. International Conference on Learning Representations (ICLR), 2014. arXiv
- G. Papamakarios, E. Nalisnick, D. J. Rezende, S. Mohamed and B. Lakshminarayanan, “Normalizing Flows for Probabilistic Modeling and Inference,” Journal of Machine Learning Research, vol. 22, no. 57, pp. 1–64, 2021. arXiv
- D. M. Blei, A. Kucukelbir and J. D. McAuliffe, “Variational Inference: A Review for Statisticians,” Journal of the American Statistical Association, vol. 112, no. 518, pp. 859–877, 2017. DOI
- C. Guo, G. Pleiss, Y. Sun and K. Q. Weinberger, “On Calibration of Modern Neural Networks,” ICML, 2017. arXiv
- C. Wang, “Calibration in Deep Learning: A Survey of the State-of-the-Art,” arXiv:2308.01222, 2023 (revised 2025). arXiv
- M. Abdar et al., “A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges,” Information Fusion, vol. 76, pp. 243–297, 2021. DOI
- J. Gawlikowski et al., “A Survey of Uncertainty in Deep Neural Networks,” Artificial Intelligence Review, vol. 56, pp. 1513–1589, 2023. DOI
- B. Lakshminarayanan, A. Pritzel and C. Blundell, “Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles,” NeurIPS (NIPS), 2017. arXiv
- Y. Ovadia et al., “Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift,” NeurIPS, 2019. arXiv
- A. N. Angelopoulos and S. Bates, “A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification,” arXiv:2107.07511, 2021. arXiv
- A. van den Oord, Y. Li and O. Vinyals, “Representation Learning with Contrastive Predictive Coding,” arXiv:1807.03748, 2018. arXiv
- B. Poole, S. Ozair, A. van den Oord, A. A. Alemi and G. Tucker, “On Variational Bounds of Mutual Information,” ICML, 2019. arXiv
- M. Tschannen, J. Djolonga, P. K. Rubenstein, S. Gelly and M. Lucic, “On Mutual Information Maximization for Representation Learning,” ICLR, 2020. arXiv