Shannon — A Mathematical Theory of Communication
Optional · paper guide 1 of 1 for ML 01B · Probability and Information Theory. Citation numbers refer to the reference list at the bottom of this page.
Navigate: this page and this track: Navigate: this page and this track
Question and thesis
How much information can a source produce, and how reliably can a channel carry it? Shannon formulates communication as recovering a message selected from possible messages, deliberately separating this engineering question from semantic meaning [5]. For ML readers, the contribution is a language for uncertainty, dependence and coding cost. The paper does not propose a classifier or claim that a high-entropy representation is automatically useful for prediction.
Method and figure

For a discrete random variable with possible values indexed by and probabilities , entropy in bits is
The sum ranges over outcomes; the convention is defined by a limit. Shannon motivates this form through continuity, increasing uncertainty for more equally likely alternatives, and consistency when a choice is split into successive choices. The logarithm’s base sets the unit, rather than changing the underlying comparison.
Read Figure 1 from left to right. The transmitter changes the representation of the message; the channel can corrupt the transmitted signal; the receiver attempts recovery. Source uncertainty and channel noise are therefore distinct objects. Conditional entropy measures uncertainty remaining after another variable is known. Repeated or predictable symbols need not carry as much new information as an independent uniform symbol from the same alphabet.
Evidence and reasoning
Theorem 2 derives the entropy form under the stated requirements. The coding theorems then connect the definitions to operational limits. Theorem 9 relates a discrete source’s entropy in bits per symbol to a noiseless channel’s capacity in bits per second. The noisy-channel argument later establishes existence of coding systems with arbitrarily small error under the relevant rate conditions. These are mathematical claims with assumptions and asymptotic constructions, not benchmark accuracies from a learned network.
Design analysis
Units and the source model matter. Bits per symbol and bits per second cannot be compared without a symbol rate. Likewise, marginal symbol entropy is not generally the entropy rate of a dependent sequence. The proof strategy reasons over long message blocks and possible codes; it does not promise a short, cheap encoder that reaches the bound for every finite message.
For learning, cross-entropy can be read as the expected cost of using a model’s coding probabilities. That connection helps explain likelihood losses, but fitting a training estimate still leaves generalization and probability calibration to be evaluated separately.
Reading checkpoint
Compare a source that always emits the same symbol with a fair binary source. Substitute their probabilities into the entropy expression and identify which uncertainty the numbers describe. Then explain why adding independent noise can increase uncertainty without adding useful information about a target label.
Limits and next step
This guide covers the discrete uncertainty and coding arguments, not every continuous-channel result in the corrected reprint. Shannon explicitly brackets semantics; entropy alone cannot rank the truth, usefulness or meaning of messages. A learned estimator also has finite-data and modeling errors absent from the ideal definition. Continue to the chapter’s cross-entropy and KL sections, keeping the true distribution, fitted distribution and empirical sample average separate.
Continue reading: ML 01B · Return to the chapter
References
- I. Goodfellow, Y. Bengio and A. Courville, Deep Learning, MIT Press, 2016, ch. 3 “Probability and Information Theory”. Book
- C. M. Bishop and H. Bishop, Deep Learning: Foundations and Concepts, Springer, 2024. Book
- K. P. Murphy, Probabilistic Machine Learning: An Introduction, MIT Press, 2022. Book
- M. P. Deisenroth, A. A. Faisal and C. S. Ong, Mathematics for Machine Learning, Cambridge University Press, 2020. Book
- C. E. Shannon, “A Mathematical Theory of Communication,” Bell System Technical Journal, vol. 27, pp. 379–423, 1948. DOI
- S. Kullback and R. A. Leibler, “On Information and Sufficiency,” The Annals of Mathematical Statistics, vol. 22, pp. 79–86, 1951. DOI
- T. M. Cover and J. A. Thomas, Elements of Information Theory, 2nd ed., Wiley, 2006. DOI
- D. P. Kingma and M. Welling, “Auto-Encoding Variational Bayes,” in Proc. International Conference on Learning Representations (ICLR), 2014. arXiv
- G. Papamakarios, E. Nalisnick, D. J. Rezende, S. Mohamed and B. Lakshminarayanan, “Normalizing Flows for Probabilistic Modeling and Inference,” Journal of Machine Learning Research, vol. 22, no. 57, pp. 1–64, 2021. arXiv
- D. M. Blei, A. Kucukelbir and J. D. McAuliffe, “Variational Inference: A Review for Statisticians,” Journal of the American Statistical Association, vol. 112, no. 518, pp. 859–877, 2017. DOI
- C. Guo, G. Pleiss, Y. Sun and K. Q. Weinberger, “On Calibration of Modern Neural Networks,” ICML, 2017. arXiv
- C. Wang, “Calibration in Deep Learning: A Survey of the State-of-the-Art,” arXiv:2308.01222, 2023 (revised 2025). arXiv
- M. Abdar et al., “A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges,” Information Fusion, vol. 76, pp. 243–297, 2021. DOI
- J. Gawlikowski et al., “A Survey of Uncertainty in Deep Neural Networks,” Artificial Intelligence Review, vol. 56, pp. 1513–1589, 2023. DOI
- B. Lakshminarayanan, A. Pritzel and C. Blundell, “Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles,” NeurIPS (NIPS), 2017. arXiv
- Y. Ovadia et al., “Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift,” NeurIPS, 2019. arXiv
- A. N. Angelopoulos and S. Bates, “A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification,” arXiv:2107.07511, 2021. arXiv
- A. van den Oord, Y. Li and O. Vinyals, “Representation Learning with Contrastive Predictive Coding,” arXiv:1807.03748, 2018. arXiv
- B. Poole, S. Ozair, A. van den Oord, A. A. Alemi and G. Tucker, “On Variational Bounds of Mutual Information,” ICML, 2019. arXiv
- M. Tschannen, J. Djolonga, P. K. Rubenstein, S. Gelly and M. Lucic, “On Mutual Information Maximization for Representation Learning,” ICLR, 2020. arXiv