A likely block is not a typical set
IntermediateIT 02 · Typical Sequences, AEP and Entropy RatesInteractive plot
Description
Compare the probability mass of a typical set with its fraction of all possible binary blocks.
Try this: Fix p=0.2 and increase block length. Does the typical set contain most probability while using most of all binary strings?
Lab · A likely block is not a typical set
- Typical probability mass
- 0.6575
- Fraction of all strings
- 0.03815
- H(p) (bits)
- 0.722
Typicality concerns information per symbol. A single likely string and a set carrying most probability are different objects.
Try this: Fix p=0.2 and increase block length. Does the typical set contain most probability while using most of all binary strings?
Independent Bernoulli(p) blocks of length n. A block is typical when |−log₂ P(block)/n−H(p)|≤ε. Each bar aggregates all C(n,k) strings with k ones. Small n can give an empty typical set; AEP is asymptotic.