How much information survives noise?
BeginnerIT 01 · Entropy, Relative Entropy and Mutual InformationInteractive plot
Description
Separate source uncertainty, transmitted information and channel capacity.
Try this: With crossover p = 0.1, move the input prior away from 0.5. Does the channel capacity change? Does the transmitted information change?
Hints
The chapter section that uses this lab:
Explanation
Answer the prediction first, or open the explanation when you are ready.
Show the explanation
Capacity is the largest I(X;Y) over all input distributions, so it is a property of the channel (here 1 − h₂(p)). Moving q lowers I(X;Y) below C; it does not change C.
Lab · How much information survives noise?
- H(X)
- 1
- H(Y)
- 1
- I(X;Y)
- 0.531
- Capacity 1 − h₂(p)
- 0.531
The output has P(Y=1) = 0.5. I(X;Y) = H(Y) − h₂(p) = 1 − 0.469 = 0.531. Random output alone does not imply useful information: at p = 0.5 the output is independent of the input.
Try this: With crossover p = 0.1, move the input prior away from 0.5. Does the channel capacity change? Does the transmitted information change?
Independent bit flips define a binary symmetric channel. q = P(X=1), p = P(Y≠X), and P(Y=1) = p + q(1−2p). All values are bits per channel use. Capacity maximizes mutual information over q; here q = 0.5 achieves it.