Full search page →

How much information survives noise?

BeginnerIT 01 · Entropy, Relative Entropy and Mutual InformationInteractive plot

Description

Separate source uncertainty, transmitted information and channel capacity.

Try this: With crossover p = 0.1, move the input prior away from 0.5. Does the channel capacity change? Does the transmitted information change?

Predict first

Move the input probability q = P(X=1) away from 0.5 with the flip probability fixed. What happens to the channel capacity C?

Explanation

Answer the prediction first, or open the explanation when you are ready.

Show the explanation

Capacity is the largest I(X;Y) over all input distributions, so it is a property of the channel (here 1 − h₂(p)). Moving q lowers I(X;Y) below C; it does not change C.

Lab · How much information survives noise?

Predict first

Move the input probability q = P(X=1) away from 0.5 with the flip probability fixed. What happens to the channel capacity C?

H(X)
1
H(Y)
1
I(X;Y)
0.531
Capacity 1 − h₂(p)
0.531

The output has P(Y=1) = 0.5. I(X;Y) = H(Y) − h₂(p) = 1 − 0.469 = 0.531. Random output alone does not imply useful information: at p = 0.5 the output is independent of the input.

Try this: With crossover p = 0.1, move the input prior away from 0.5. Does the channel capacity change? Does the transmitted information change?

Independent bit flips define a binary symmetric channel. q = P(X=1), p = P(Y≠X), and P(Y=1) = p + q(1−2p). All values are bits per channel use. Capacity maximizes mutual information over q; here q = 0.5 achieves it.

Method sources

Open in the Playground →