Full search page →

Shannon — A Mathematical Theory of Communication

Optional · paper guide 1 of 1 for ML 01B · Probability and Information Theory. Citation numbers refer to the reference list at the bottom of this page.

Navigate: this page and this track: Navigate: this page and this track

Question and thesis

How much information can a source produce, and how reliably can a channel carry it? Shannon formulates communication as recovering a message selected from possible messages, deliberately separating this engineering question from semantic meaning [5]. For ML readers, the contribution is a language for uncertainty, dependence and coding cost. The paper does not propose a classifier or claim that a high-entropy representation is automatically useful for prediction.

Method and figure

The source, transmitter, noisy channel, receiver and destination have separate roles. Entropy describes uncertainty of messages, not their semantic importance.
Source: Shannon 1948; corrected reprint of both Bell System Technical Journal parts, hosted at Harvard. Figure 1, PDF p. 2. The source, transmitter, noisy channel, receiver and destination have separate roles. Entropy describes uncertainty of messages, not their semantic importance. [5]

For a discrete random variable XX with possible values indexed by ii and probabilities pip_i, entropy in bits is

H(X)=−∑ipilog⁡2pi.H(X)=-\sum_i p_i\log_2 p_i.

The sum ranges over outcomes; the convention 0log⁡20=00\log_2 0=0 is defined by a limit. Shannon motivates this form through continuity, increasing uncertainty for more equally likely alternatives, and consistency when a choice is split into successive choices. The logarithm’s base sets the unit, rather than changing the underlying comparison.

Read Figure 1 from left to right. The transmitter changes the representation of the message; the channel can corrupt the transmitted signal; the receiver attempts recovery. Source uncertainty and channel noise are therefore distinct objects. Conditional entropy measures uncertainty remaining after another variable is known. Repeated or predictable symbols need not carry as much new information as an independent uniform symbol from the same alphabet.

Evidence and reasoning

Theorem 2 derives the entropy form under the stated requirements. The coding theorems then connect the definitions to operational limits. Theorem 9 relates a discrete source’s entropy in bits per symbol to a noiseless channel’s capacity in bits per second. The noisy-channel argument later establishes existence of coding systems with arbitrarily small error under the relevant rate conditions. These are mathematical claims with assumptions and asymptotic constructions, not benchmark accuracies from a learned network.

Design analysis

Units and the source model matter. Bits per symbol and bits per second cannot be compared without a symbol rate. Likewise, marginal symbol entropy is not generally the entropy rate of a dependent sequence. The proof strategy reasons over long message blocks and possible codes; it does not promise a short, cheap encoder that reaches the bound for every finite message.

For learning, cross-entropy can be read as the expected cost of using a model’s coding probabilities. That connection helps explain likelihood losses, but fitting a training estimate still leaves generalization and probability calibration to be evaluated separately.

Reading checkpoint

Compare a source that always emits the same symbol with a fair binary source. Substitute their probabilities into the entropy expression and identify which uncertainty the numbers describe. Then explain why adding independent noise can increase uncertainty without adding useful information about a target label.

Limits and next step

This guide covers the discrete uncertainty and coding arguments, not every continuous-channel result in the corrected reprint. Shannon explicitly brackets semantics; entropy alone cannot rank the truth, usefulness or meaning of messages. A learned estimator also has finite-data and modeling errors absent from the ideal definition. Continue to the chapter’s cross-entropy and KL sections, keeping the true distribution, fitted distribution and empirical sample average separate.

Continue reading: ML 01B · Return to the chapter

References

  1. I. Goodfellow, Y. Bengio and A. Courville, Deep Learning, MIT Press, 2016, ch. 3 “Probability and Information Theory”. Book
  2. C. M. Bishop and H. Bishop, Deep Learning: Foundations and Concepts, Springer, 2024. Book
  3. K. P. Murphy, Probabilistic Machine Learning: An Introduction, MIT Press, 2022. Book
  4. M. P. Deisenroth, A. A. Faisal and C. S. Ong, Mathematics for Machine Learning, Cambridge University Press, 2020. Book
  5. C. E. Shannon, “A Mathematical Theory of Communication,” Bell System Technical Journal, vol. 27, pp. 379–423, 1948. DOI
  6. S. Kullback and R. A. Leibler, “On Information and Sufficiency,” The Annals of Mathematical Statistics, vol. 22, pp. 79–86, 1951. DOI
  7. T. M. Cover and J. A. Thomas, Elements of Information Theory, 2nd ed., Wiley, 2006. DOI
  8. D. P. Kingma and M. Welling, “Auto-Encoding Variational Bayes,” in Proc. International Conference on Learning Representations (ICLR), 2014. arXiv
  9. G. Papamakarios, E. Nalisnick, D. J. Rezende, S. Mohamed and B. Lakshminarayanan, “Normalizing Flows for Probabilistic Modeling and Inference,” Journal of Machine Learning Research, vol. 22, no. 57, pp. 1–64, 2021. arXiv
  10. D. M. Blei, A. Kucukelbir and J. D. McAuliffe, “Variational Inference: A Review for Statisticians,” Journal of the American Statistical Association, vol. 112, no. 518, pp. 859–877, 2017. DOI
  11. C. Guo, G. Pleiss, Y. Sun and K. Q. Weinberger, “On Calibration of Modern Neural Networks,” ICML, 2017. arXiv
  12. C. Wang, “Calibration in Deep Learning: A Survey of the State-of-the-Art,” arXiv:2308.01222, 2023 (revised 2025). arXiv
  13. M. Abdar et al., “A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges,” Information Fusion, vol. 76, pp. 243–297, 2021. DOI
  14. J. Gawlikowski et al., “A Survey of Uncertainty in Deep Neural Networks,” Artificial Intelligence Review, vol. 56, pp. 1513–1589, 2023. DOI
  15. B. Lakshminarayanan, A. Pritzel and C. Blundell, “Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles,” NeurIPS (NIPS), 2017. arXiv
  16. Y. Ovadia et al., “Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift,” NeurIPS, 2019. arXiv
  17. A. N. Angelopoulos and S. Bates, “A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification,” arXiv:2107.07511, 2021. arXiv
  18. A. van den Oord, Y. Li and O. Vinyals, “Representation Learning with Contrastive Predictive Coding,” arXiv:1807.03748, 2018. arXiv
  19. B. Poole, S. Ozair, A. van den Oord, A. A. Alemi and G. Tucker, “On Variational Bounds of Mutual Information,” ICML, 2019. arXiv
  20. M. Tschannen, J. Djolonga, P. K. Rubenstein, S. Gelly and M. Lucic, “On Mutual Information Maximization for Representation Learning,” ICLR, 2020. arXiv