Full search page →

Separate entropy from model mismatch

BeginnerML 01B · Probability and Information TheoryInteractive plot

Description

Change the true probability and the model probability; measure the extra coding cost caused by a wrong model.

Try this: Fix p=0.8. Find the q that minimizes cross-entropy, then compare q=0.2.

Lab · Separate entropy from model mismatch

Entropy
0.722
Cross-entropy
1
KL divergence
0.278

H(P,Q) = 0.722 + 0.278 bits. Matching probabilities removes the KL penalty, but cannot remove source uncertainty.

Try this: Fix p=0.8. Find the q that minimizes cross-entropy, then compare q=0.2.

For Bernoulli sources, H(p,q)=−p log₂q−(1−p)log₂(1−q)=H(p)+D(p∥q). q stays strictly between 0 and 1. These are expected ideal bit costs, not measured file sizes.

Method sources

Open in the Playground →