Separate entropy from model mismatch
BeginnerML 01B · Probability and Information TheoryInteractive plot
Description
Change the true probability and the model probability; measure the extra coding cost caused by a wrong model.
Try this: Fix p=0.8. Find the q that minimizes cross-entropy, then compare q=0.2.
Hints
The chapter section that uses this lab:
Lab · Separate entropy from model mismatch
- Entropy
- 0.722
- Cross-entropy
- 1
- KL divergence
- 0.278
H(P,Q) = 0.722 + 0.278 bits. Matching probabilities removes the KL penalty, but cannot remove source uncertainty.
Try this: Fix p=0.8. Find the q that minimizes cross-entropy, then compare q=0.2.
For Bernoulli sources, H(p,q)=−p log₂q−(1−p)log₂(1−q)=H(p)+D(p∥q). q stays strictly between 0 and 1. These are expected ideal bit costs, not measured file sizes.