Full search page →

IT 07 · Statistical Evidence, Maximum Entropy and Growth

Information TheoryAdvanced18 minUpdated Nov 5, 2026

Needs: IT 06 · Rate–Distortion and Lossy Representations

Navigate: this page and this track: Navigate: this page and this track
Explore this chapter 1 interactive experiment

Slide a decision threshold across two exact binomial distributions and compute both error probabilities.

What you’ll learn

  • Interpret a likelihood ratio as evidence rather than a posterior probability.
  • Use empirical types to explain divergence-controlled rare-event rates.
  • Distinguish hypothesis-testing error tradeoffs from parameter estimation.
  • Derive Fisher information and a regular unbiased-estimation bound.
  • Construct maximum-entropy distributions from explicit constraints.
  • Relate log-optimal allocation and universal mixtures to coding regret.

The big picture

Information measures are not restricted to storage and communication. They also quantify how observations distinguish competing explanations, how strongly a distribution changes with a parameter, and how costly a poor predictive decision can become over repeated trials.

The book map is Cover and Thomas’s Chapter 11 for statistics, Chapter 12 for maximum entropy, Chapter 6 for gambling and coding, and Chapter 16 for portfolio theory [1]. Stanford’s method-of-types notes [2] provide an accessible primary finite-alphabet derivation.

We use mathematical decision games with fully specified probabilities and payoffs. Their purpose is to expose information identities, not to infer practical returns from historical data. In every section, distinguish the source law, the chosen model, the observed sample and the decision criterion.

Likelihood ratios compare explanations

A sample supports one model over another when that model assigns the sample a larger probability. Repeated independent observations add their log evidence.

For two fully specified distributions PP and QQ with compatible support, define

Λ(xn)=log⁡P(xn)Q(xn)=∑t=1nlog⁡P(xt)Q(xt).\Lambda(x^n)=\log\frac{P(x^n)}{Q(x^n)} =\sum_{t=1}^{n}\log\frac{P(x_t)}{Q(x_t)}.

Here xnx^n is an observed length-nn sample, the final equality assumes independent observations under each model, and log⁡\log may be natural or base two if used consistently. Positive values favor PP in the likelihood comparison.

Under PP, expected base-two log ratio is nDKL(P∥Q)nD_{\mathrm{KL}}(P\Vert Q). Under QQ, it is −nDKL(Q∥P)-nD_{\mathrm{KL}}(Q\Vert P). The different directions arise because expectation is taken under different laws.

A likelihood ratio is not a posterior probability. To calculate posterior odds, multiply by prior odds. To choose an action, incorporate the losses of wrong decisions. A threshold of zero is appropriate for certain symmetric prior-and-loss choices, not every test.

For simple hypotheses, the Neyman–Pearson likelihood-ratio principle describes the most powerful test at a specified false-alarm constraint [1]. “Simple” means the distributions are fixed, not families with unknown fitted parameters.

If parameters are estimated from the same sample, the test is a different procedure. Model selection, nuisance parameters and threshold calibration can alter its finite error behavior. It is useful to start with fully specified toy laws before interpreting a fitted classifier’s score as evidence.

Testing errors form a tradeoff

Making it harder to declare an alternative reduces false alarms but can miss more real alternatives. Neither error should disappear from the report.

Set null distribution P0P_0 and alternative P1P_1. A rejection region AA gives

α=P0(Xn∈A),β=P1(Xn∉A).\alpha=P_0(X^n\in A),\qquad \beta=P_1(X^n\notin A).

Here α\alpha is the type-I error or false-alarm probability, β\beta the type-II error or missed-alternative probability, and AA the set where the test declares the alternative.

For Bernoulli probabilities p0<p1p_0<p_1, the log likelihood ratio of alternative to null is an increasing function of the count of ones. A threshold on that count therefore implements a likelihood-ratio test.

Discrete counts may prevent reaching an arbitrary false-alarm target exactly with a deterministic threshold. Randomization at a boundary can achieve an intermediate operating point mathematically; a deployed rule should state whether it uses such randomization.

For 30 Bernoulli observations under probabilities 0.3 and 0.6, count thresholds trade false-positive probability against detection probability.
Figure 1 — Exact binomial probabilities. Detection probability is one minus type-II error; each point corresponds to a stated count threshold.

Code: exact errors and a Monte Carlo check

This complete example chooses the first count threshold satisfying a 5% false-alarm limit and calculates its missed-alternative probability. It then verifies the two laws independently.

Code · Python · 17 lines
import numpy as np
from scipy.stats import binom

n, p0, p1 = 30, 0.3, 0.6
thresholds = np.arange(n+2)
false_alarm = binom.sf(thresholds-1, n, p0)
detection = binom.sf(thresholds-1, n, p1)
threshold = thresholds[np.flatnonzero(false_alarm <= 0.05)[0]]
alpha = binom.sf(threshold-1, n, p0)
beta = binom.cdf(threshold-1, n, p1)
rng = np.random.default_rng(40)
observed_alpha = np.mean(rng.binomial(n, p0, 100000) >= threshold)
observed_beta = np.mean(rng.binomial(n, p1, 100000) < threshold)
print("Threshold / alpha / beta:", threshold, alpha, beta)
print("Simulation alpha / beta:", observed_alpha, observed_beta)
assert abs(observed_alpha-alpha) < 0.005
assert abs(observed_beta-beta) < 0.005

The selected threshold uses known distributions, not a test-set search. In a learned application, tune the threshold on validation data and estimate its errors on separate test observations. Data groups or time dependence can change the effective independence assumption.

The proportion of alternatives in an evaluation set affects the frequency of correct declarations, but it does not change the two conditional errors for a fixed test and fixed laws. A test can have a small false-alarm probability and still produce many false declarations when alternatives are extremely rare. To interpret those declarations, combine the conditional errors with the intended prior prevalence. This is a direct consequence of conditional probability, not a failure of the likelihood-ratio principle. State whether a reported quantity is conditioned on the null, conditioned on the alternative, or conditioned on the test’s output.

Worked evidence is not the same as confidence

For the example’s null probability 0.3 and alternative probability 0.6, a length-30 block with kk ones has alternative-to-null log likelihood ratio

Λk=kln⁡0.60.3+(30−k)ln⁡0.40.7.\Lambda_k=k\ln\frac{0.6}{0.3}+(30-k)\ln\frac{0.4}{0.7}.

Here kk is the observed count, Λk\Lambda_k uses natural logarithms, and the two fractions compare the probabilities of one and zero under the competing laws. With 18 ones, Λ18≈5.761260\Lambda_{18}\approx5.761260, giving likelihood ratio about 317.748375. This evidence favors the alternative strongly.

If the prior alternative probability is 0.01, prior odds are 0.01/0.990.01/0.99. Posterior odds become approximately 317.748375×0.01/0.99=3.209580317.748375\times0.01/0.99=3.209580, corresponding to posterior probability about 0.762447. The likelihood ratio is not a 99.7% confidence value: the prior changes the posterior interpretation.

An even more practical distinction concerns all positive declarations from the threshold test. The code chooses threshold 14, with false-alarm probability about 0.040053 and missed-alternative probability about 0.048112. If only 1% of cases come from the alternative, the probability that a positive declaration is truly alternative is

Pr⁡(H1∣positive)=π(1−β)π(1−β)+(1−π)α≈0.193588.\Pr(H_1\mid\text{positive}) =\frac{\pi(1-\beta)}{\pi(1-\beta)+(1-\pi)\alpha} \approx0.193588.

Here π=0.01\pi=0.01 is prior alternative prevalence, α\alpha the test’s false-alarm probability and β\beta its missed-alternative probability. The notation H1H_1 denotes the event that the alternative generated the case. Most positives in this population are false alarms, even though each conditional error is below 5%.

This conclusion follows from counting expected positive cases from the two populations. It does not change the likelihood-ratio optimality result or the measured conditional errors. A decision report should name its conditioning event so readers can distinguish a particular sample’s posterior from the average composition of all positive declarations.

Code · Python · 14 lines
import numpy as np
from scipy.stats import binom

n, p0, p1, prior = 30, 0.3, 0.6, 0.01
count = 18
log_ratio = count*np.log(p1/p0)+(n-count)*np.log((1-p1)/(1-p0))
posterior = 1/(1+((1-prior)/prior)*np.exp(-log_ratio))
threshold = 14
alpha = binom.sf(threshold-1, n, p0)
beta = binom.cdf(threshold-1, n, p1)
positive_precision = prior*(1-beta)/(prior*(1-beta)+(1-prior)*alpha)
assert alpha <= 0.05 and binom.sf(threshold-2, n, p0) > 0.05
print("Likelihood ratio / sample posterior:", np.exp(log_ratio), posterior)
print("Probability alternative given positive:", positive_precision)

Empirical types reveal why divergence controls rare events

Many strings share the same histogram. Their number grows like an entropy exponent, while their individual source probabilities shrink like a cross-entropy exponent. The difference is divergence.

For a finite alphabet of size kk, the type p^\widehat p of a length-nn sequence is its empirical symbol distribution. Its type class T(p^)T(\widehat p) contains every sequence with those counts. Under independent source law qq,

q(xn)=2−n[H(p^)+DKL(p^∥q)].q(x^n)=2^{-n[H(\widehat p)+D_{\mathrm{KL}}(\widehat p\Vert q)]}.

Here xnx^n has type p^\widehat p, H(p^)H(\widehat p) is empirical entropy, and divergence is measured in bits. The formula follows by grouping identical log probabilities by count.

The class size is bounded, with a conservative polynomial factor, by

(n+1)−k2nH(p^)≤∣T(p^)∣≤2nH(p^).(n+1)^{-k}2^{nH(\widehat p)} \leq |T(\widehat p)|\leq2^{nH(\widehat p)}.

Here ∣T∣|T| is the number of sequences; nn and kk are block length and alphabet size. Multiplying by one sequence probability gives

(n+1)−k2−nDKL(p^∥q)≤q(T(p^))≤2−nDKL(p^∥q).(n+1)^{-k}2^{-nD_{\mathrm{KL}}(\widehat p\Vert q)} \leq q(T(\widehat p)) \leq2^{-nD_{\mathrm{KL}}(\widehat p\Vert q)}.

Here q(T)q(T) is total type-class probability. The polynomial factor matters at finite length but disappears from the normalized logarithmic exponent [1,2].

The orientation is worth emphasizing: the first argument is the unusual empirical type; the second is the source actually generating observations. This differs from a coding mismatch calculation averaged under a fixed true source.

For sets of empirical distributions, the dominant exponential behavior is governed by the smallest divergence among allowed types. Sanov’s theorem formalizes that large-deviation statement under its finite-alphabet and set-regularity conditions [1].

Code: exact binary type probability versus its exponent

This standalone example compares a binomial probability with the exponential bounds. It uses logarithms to avoid underflow and checks the finite polynomial gap.

Code · Python · 15 lines
import numpy as np
from scipy.stats import binom

q, target = 0.3, 0.6
for n in [30, 100, 300, 1000]:
    count = int(round(n*target))
    empirical = count/n
    divergence = (empirical*np.log2(empirical/q) +
                  (1-empirical)*np.log2((1-empirical)/(1-q)))
    negative_log_probability = -binom.logpmf(count, n, q)/np.log(2)
    exponent = n*divergence
    assert exponent <= negative_log_probability + 1e-10
    assert negative_log_probability <= exponent + 2*np.log2(n+1) + 1e-10
    print("Length / observed normalized exponent / divergence:",
          n, negative_log_probability/n, divergence)

A rare exact type is not the same event as a one-sided tail or a typical neighborhood. The exponent governing a tail comes from optimizing over all counts in that event, not blindly substituting its most extreme count.

From one type to a set of types

For an event defined by a collection A\mathcal A of empirical distributions, there are at most (n+1)k(n+1)^k possible types. Summing the individual upper bounds gives

q(p^∈A)≤(n+1)k2−ninf⁡v∈AnDKL(v∥q).q(\widehat p\in\mathcal A) \leq(n+1)^k2^{-n\inf_{v\in\mathcal A_n}D_{\mathrm{KL}}(v\Vert q)}.

Here An\mathcal A_n is the subset of A\mathcal A realizable at length nn, vv is a candidate type, and the infimum selects the least surprising type under source qq. An empty event has probability zero and is treated separately.

The most probable portion of a rare event often lies at its boundary closest to the source, rather than at its most dramatic extreme. For a binary source with probability 0.3, a tail requiring frequency at least 0.6 is governed near frequency 0.6, not frequency one.

Finite type grids matter: the required boundary may fall between attainable fractions. Exact binomial sums handle that discreteness. The asymptotic divergence picture explains the trend while leaving rounding and polynomial factors visible.

This viewpoint is useful for interpreting apparent distribution shifts. A statistically unusual histogram can be evidence against a source law, but it does not identify why the law failed. Dependence, selection bias, measurement changes and a different source can all alter observed frequencies.

Asymmetric and symmetric hypothesis exponents

Different ways of budgeting the two testing errors lead to different information limits. Divergence is one such limit, not the answer to every classification problem.

Suppose observations are independent under null PP or alternative QQ, and type-I error is kept below a fixed nontrivial tolerance. The optimal type-II exponent approaches

−1nln⁡βn→Dnats(P∥Q).-\frac1n\ln\beta_n\to D_{\mathrm{nats}}(P\Vert Q).

Here βn\beta_n is alternative probability of incorrectly accepting the null, and DnatsD_{\mathrm{nats}} is divergence with natural logarithms. This is the classical Stein asymptotic regime under suitable source/support conditions [1].

The direction follows the acceptance region: typical null observations must remain accepted, and the alternative probability of looking null has a PP-to-QQ divergence exponent. Relabeling hypotheses reverses the divergence and can change the exponent.

With fixed nonzero priors and a symmetric Bayesian error objective, the exponent instead involves Chernoff information:

CCh(P,Q)=−min⁡0≤s≤1ln⁡∑xP(x)sQ(x)1−s.C_{\mathrm{Ch}}(P,Q) =-\min_{0\leq s\leq1}\ln\sum_xP(x)^sQ(x)^{1-s}.

Here ss interpolates between the two laws and xx ranges over their finite alphabet. This quantity is symmetric despite the asymmetric appearance of the summand.

These exponents describe large independent samples, not exact errors for a short sample. The binomial experiment above is a direct finite calculation. Using an asymptotic exponent as an exact probability can be inaccurate when prefactors and decision boundaries matter.

Composite hypotheses, dependence, changing populations and unequal losses require their own analysis. An information-theoretic limit can explain which source differences help discrimination while leaving practical calibration and sampling design unresolved.

Sufficiency and data processing in inference

A summary is sufficient for a parameter when it retains all the sample’s evidence about that parameter within the specified model family.

For a family pθ(xn)p_\theta(x^n), a statistic T(Xn)T(X^n) is sufficient when the conditional sample law given TT does not depend on θ\theta. A factorization criterion expresses this as

pθ(xn)=gθ(T(xn))h(xn).p_\theta(x^n)=g_\theta(T(x^n))h(x^n).

Here θ\theta is the parameter, TT the statistic, gθg_\theta the parameter-dependent factor and hh a parameter-independent factor, under the relevant family and domination assumptions [1].

For independent Bernoulli trials, the number of ones is sufficient for the unknown probability. The ordering carries no extra parameter evidence in that family. For a Markov family, the ordering can matter; a single count of ones is no longer generally sufficient.

If a random parameter Θ\Theta has a specified prior, sufficiency gives the conditional relation needed for I(Θ;Xn)=I(Θ;T(Xn))I(\Theta;X^n)=I(\Theta;T(X^n)). Arbitrary summaries satisfy only the data-processing upper bound.

The qualification “within the family” matters. A statistic sufficient for one assumed model may discard evidence that the model is wrong. A mean can estimate a location while hiding multimodality, dependence or an outlier pattern.

This connects to learned representations: retaining information about one target is different from retaining every diagnostic property of the raw data. The task and source family must accompany a sufficiency claim.

Fisher information measures local parameter sensitivity

Fisher information measures how strongly the log probability changes when a parameter moves slightly. It is a local property of a model, distinct from mutual information between random variables.

For a scalar parameter θ\theta and regular density or mass function pθp_\theta, define score and information:

sθ(X)=∂∂θln⁡pθ(X),J(θ)=Eθ[sθ(X)2].s_\theta(X)=\frac{\partial}{\partial\theta}\ln p_\theta(X), \qquad \mathcal J(\theta)=\mathbb E_\theta[s_\theta(X)^2].

Here sθs_\theta is the score, J\mathcal J Fisher information per observation, and expectation is under the parameterized model. Under regularity, score expectation is zero.

For nn independent observations, information adds to nJ(θ)n\mathcal J(\theta). The Cramér–Rao inequality for a regular unbiased estimator θ^\widehat\theta gives

Var⁡θ(θ^)≥1nJ(θ).\operatorname{Var}_\theta(\widehat\theta)\geq \frac1{n\mathcal J(\theta)}.

Here variance describes repeated samples at the fixed parameter. The bound requires differentiability, exchange of differentiation and integration, and appropriate support and unbiasedness assumptions [1].

For Bernoulli probability θ∈(0,1)\theta\in(0,1), J(θ)=1/[θ(1−θ)]\mathcal J(\theta)=1/[\theta(1-\theta)]. The sample mean is unbiased with variance θ(1−θ)/n\theta(1-\theta)/n, achieving that bound.

The information changes with parameter coordinates. Reparameterizing a probability as a log odds multiplies the information by the squared parameter derivative. This is compatible with an unchanged underlying statistical family; the units of parameter error have changed.

Irregular models with parameter-dependent support or biased estimators need different bounds. A small empirical estimator variance alone does not prove accuracy if its mean is displaced from the parameter.

Optional: When the support moves with the parameter (in the Supplementary notes). A uniform-source example shows exactly which regularity identity fails.

Derive the Bernoulli information and variance bound

For one Bernoulli observation x∈{0,1}x\in\{0,1\}, log likelihood is xln⁡θ+(1−x)ln⁡(1−θ)x\ln\theta+(1-x)\ln(1-\theta). Differentiating gives

sθ(x)=xθ−1−x1−θ=x−θθ(1−θ).s_\theta(x)=\frac{x}{\theta}-\frac{1-x}{1-\theta} =\frac{x-\theta}{\theta(1-\theta)}.

Here θ∈(0,1)\theta\in(0,1) is the probability of one and sθs_\theta the score. Its two possible values are 1/θ1/\theta and −1/(1−θ)-1/(1-\theta). Averaging their squares with probabilities θ\theta and 1−θ1-\theta yields J(θ)=1/[θ(1−θ)]\mathcal J(\theta)=1/[\theta(1-\theta)].

Why does its reciprocal bound estimation variance? Let TT be a regular unbiased estimator based on nn independent observations, and let SnS_n be the sum of their scores. Differentiate unbiasedness, EθT=θ\mathbb E_\theta T=\theta, under the regularity assumptions:

1=ddθEθT=Eθ[(T−θ)Sn].1=\frac{d}{d\theta}\mathbb E_\theta T =\mathbb E_\theta[(T-\theta)S_n].

Here TT is a fixed function of the observed sample, Sn=∑i=1nsθ(Xi)S_n=\sum_{i=1}^ns_\theta(X_i) is its total score, and EθSn=0\mathbb E_\theta S_n=0. The right side is the covariance between estimator and score. Squared covariance cannot exceed the product of variances. Since Var⁡θ(Sn)=nJ(θ)\operatorname{Var}_\theta(S_n)=n\mathcal J(\theta), the variance bound follows.

For θ=0.3\theta=0.3 and n=100n=100, information per observation is approximately 4.761905 and the variance lower bound is 0.0021. The sample mean reaches it because its centered error is proportional to the total score. Increasing independent sample size by four divides this variance by four and its standard deviation by two. Independence is what makes score variances add; repeatedly measuring the same correlated trial would not justify that scaling.

Fisher information is local divergence curvature

Under suitable twice-differentiable regularity, neighboring parameter distributions satisfy

Dnats(pθ∥pθ+δ)=12J(θ)δ2+o(δ2).D_{\mathrm{nats}}(p_\theta\Vert p_{\theta+\delta}) =\frac12\mathcal J(\theta)\delta^2+o(\delta^2).

Here δ\delta is a small parameter displacement, J(θ)\mathcal J(\theta) is the natural-log score information, and o(δ2)o(\delta^2) denotes a remainder whose ratio to δ2\delta^2 tends to zero. With base-two divergence, divide the leading term by ln⁡2\ln2.

The first derivative vanishes because the score has zero expectation. The second derivative gives the Fisher information under the regular model. This links local estimation sensitivity with the global divergence used in testing and coding.

It is a local approximation. A large parameter move can involve different curvature, changing supports or multiple modes. Treating one Fisher value as an exact distance across a whole model family would discard those differences.

Maximum entropy makes constraints explicit

If only a few expectations are known, choose the distribution with largest entropy among those matching them. The answer depends on the stated constraints and allowed support.

On a finite alphabet with features ϕj(x)\phi_j(x), impose E[ϕj(X)]=mj\mathbb E[\phi_j(X)]=m_j. Maximizing entropy with normalization gives an exponential-family form

pλ(x)=exp⁡(∑jλjϕj(x))Z(λ),Z(λ)=∑xexp⁡(∑jλjϕj(x)).p_\lambda(x)=\frac{\exp(\sum_j\lambda_j\phi_j(x))}{Z(\lambda)}, \qquad Z(\lambda)=\sum_x\exp(\sum_j\lambda_j\phi_j(x)).

Here λj\lambda_j are multipliers chosen to satisfy the moment constraints, ZZ normalizes the mass function, and the positive-sign convention is a choice of multiplier sign. Existence and interior-support conditions matter.

Differentiating ln⁡Z\ln Z gives the constrained feature expectations. The entropy optimum is not determined by a mean alone unless support is also known. On a bounded interval, fixed support without other constraints gives a uniform density; fixed real mean and variance give a Gaussian; nonnegative support and a fixed mean give an exponential density [1].

Maximum entropy is a transparent modeling principle, not evidence that the real source belongs to the chosen family. Unmeasured dependencies and higher moments remain possible. A useful analysis records constraints, fits multipliers and then tests the resulting predictions on separate data.

The exponential form also explains its relationship to divergence. If a candidate distribution matches the specified feature expectations of the maximum-entropy model, averaging the model’s log probability depends only on those shared moments. Relative entropy from the candidate to the model then equals the model entropy minus candidate entropy, using the same logarithm base.

That is another instance of a constraint turning a cross-entropy term into a fixed quantity. The Gaussian maximum-entropy proof from the previous chapter is a continuous moment-constrained example. This common algebra clarifies why maximum entropy, exponential families and information projections repeatedly appear together.

Derive and interpret the maximum-entropy distribution

Use natural logarithms during the optimization; this only rescales entropy and its multipliers. Add the normalization constraint and feature constraints to the entropy objective:

F(p)=−∑xpxln⁡px+α(∑xpx−1)+∑jλj(∑xpxϕj(x)−mj).\mathcal F(p) =-\sum_xp_x\ln p_x +\alpha\left(\sum_xp_x-1\right) +\sum_j\lambda_j\left(\sum_xp_x\phi_j(x)-m_j\right).

Here pxp_x are candidate probabilities, α\alpha is the normalization multiplier, λj\lambda_j the feature multipliers, ϕj(x)\phi_j(x) feature jj and mjm_j its required expectation. At an interior optimum, differentiating each probability gives −ln⁡px−1+α+∑jλjϕj(x)=0-\ln p_x-1+\alpha+\sum_j\lambda_j\phi_j(x)=0. Exponentiating and normalizing yields the earlier exponential form.

For states 0, 1, 2 with mean 0.7, there is one feature ϕ(x)=x\phi(x)=x. Relative weights are 1,eλ,e2λ1,e^\lambda,e^{2\lambda}. A uniform distribution would have mean one, so the desired smaller mean requires a negative multiplier. The code finds λ≈−0.466121\lambda\approx-0.466121, with probabilities approximately 0.494780, 0.310440 and 0.194780.

This model spreads probability over all allowed states while matching the mean. A point mass or a two-state mixture could satisfy some moment constraints too, but would have lower entropy here. At a feasible boundary, some probabilities can vanish and finite interior multipliers may no longer suffice. The support, moments and existence conditions are therefore part of the mathematical problem, rather than choices made after selecting a favored density.

Code: solve a finite maximum-entropy model and verify Fisher variance

This complete example finds a three-state distribution with a specified mean and compares Bernoulli estimator variance with its exact unbiased bound.

Code · Python · 19 lines
import numpy as np
from scipy.optimize import brentq
from scipy.special import softmax

states = np.array([0.0, 1.0, 2.0])
target_mean = 0.7
probability = lambda lam: softmax(lam*states)
lam = brentq(lambda value: probability(value) @ states-target_mean, -20, 20)
p = probability(lam)
assert abs(p @ states-target_mean) < 1e-12
print("Maximum-entropy multiplier / probabilities:", lam, p)

rng = np.random.default_rng(41)
theta, n = 0.3, 100
estimates = rng.binomial(n, theta, size=100000)/n
fisher = 1/(theta*(1-theta))
bound = 1/(n*fisher)
print("Empirical estimator variance / exact bound:", estimates.var(), bound)
assert abs(estimates.var()-bound) < 0.0001

The maximum-entropy calculation uses exactly the enumerated support. Allowing other states would change the optimization. The variance calculation checks a synthetic regular case, not a universal lower bound for every estimator design.

Logarithmic growth turns mismatch into a decision cost

In a repeated multiplicative game, a strategy’s errors compound. Maximizing expected logarithmic growth makes the best allocation match the outcome probabilities under the stated fair-payoff model.

Let outcome ii occur with probability pip_i. Allocate fraction qi>0q_i>0 to its contingent payoff, with ∑iqi=1\sum_iq_i=1. A fair reference law ri>0r_i>0 pays multiplier 1/ri1/r_i when outcome ii occurs. Then expected base-two log growth is

g(p,q;r)=∑ipilog⁡2qiri=DKL(p∥r)−DKL(p∥q).g(p,q;r)=\sum_ip_i\log_2\frac{q_i}{r_i} =D_{\mathrm{KL}}(p\Vert r)-D_{\mathrm{KL}}(p\Vert q).

Here pp is the outcome law, qq the allocation, rr the reciprocal-payoff probability vector, and gg is log multiplier per trial. The identity uses normalized laws and the stipulated contingent-payoff game [1].

The optimum is q=pq=p on the source support. A mismatched strategy loses exactly D(p∥q)D(p\Vert q) expected log growth. For two outcomes with equal multiplier two, rr is fair and the optimum value is 1−h2(p)1-h_2(p).

This is the Kelly connection between prediction and growth. It does not mean every portfolio has contingent fair payoffs or that probabilities are known. Adding cash, transaction costs, leverage restrictions or different payoffs changes the optimization.

Expected logarithmic growth for probability 0.65 and equal contingent multiplier two peaks when the allocation equals 0.65.
Figure 2 — Analytical two-outcome payoff model. The decrease from the optimum is probability-model divergence.

If a side observation YY is available before each outcome, optimize allocation conditionally. The improvement over a fixed optimal allocation is

EY[g(pX∣Y,pX∣Y;r)]−g(pX,pX;r)=I(X;Y).\mathbb E_Y[g(p_{X\mid Y},p_{X\mid Y};r)] -g(p_X,p_X;r)=I(X;Y).

Here XX is the outcome, YY the side information, and the reference payoff law rr is unchanged. Mutual information has an operational value in this particular log-growth game.

Universal mixtures connect portfolios with coding

Instead of committing to one fixed allocation, mix the wealth of many fixed strategies. Past outcomes then determine how much each strategy contributes.

For a positive return vector RtR_t and a fixed simplex allocation bb, cumulative wealth is

Wn(b)=∏t=1nb⊤Rt,Wnmix=∫Wn(b) π(db).W_n(b)=\prod_{t=1}^{n}b^\top R_t, \qquad W_n^{\mathrm{mix}}=\int W_n(b)\,\pi(db).

Here tt indexes trials, bb has nonnegative entries summing to one, RtR_t contains each component’s relative payoff, and π\pi is a prior measure over fixed allocations. The mixture has an equivalent sequential allocation obtained by weighting strategies by their past wealth.

Cover and Thomas’s Chapter 16 develops universal-portfolio ideas and their long-run comparisons with a best constant allocation under stated conditions [1]. The benchmark is a fixed allocation in hindsight, not an unrestricted future-knowing trader.

For the two-outcome fair contingent game, a uniform prior on the probability parameter yields the same predictive mechanism as a beta mixture. Its sequential probability of one is (N1+1)/(t+2)(N_1+1)/(t+2) after tt observations. This is a whole-count prior, distinct from the half-count predictor in lossless coding.

Code: mixture growth and the best constant allocation

This complete example computes wealth in logarithms and uses the beta integral for the uniform mixture. It compares a known source strategy with the retrospectively best fixed strategy and the causal mixture.

Code · Python · 23 lines
import numpy as np
from scipy.special import betaln

rng = np.random.default_rng(42)
p, n = 0.65, 2000
outcomes = rng.binomial(1, p, size=n)
ones, zeros = int(outcomes.sum()), n-int(outcomes.sum())
fitted = ones/n
oracle_log_wealth = n*np.log(2)+ones*np.log(p)+zeros*np.log(1-p)
best_log_wealth = n*np.log(2)+ones*np.log(fitted)+zeros*np.log(1-fitted)
mixture_log_wealth = n*np.log(2)+betaln(ones+1, zeros+1)
sequential_log_wealth, previous_ones = 0.0, 0
for t, outcome in enumerate(outcomes):
    q = (previous_ones+1)/(t+2)
    sequential_log_wealth += np.log(2*(q if outcome else 1-q))
    previous_ones += int(outcome)
assert abs(sequential_log_wealth-mixture_log_wealth) < 1e-9
assert mixture_log_wealth <= best_log_wealth
print("Source / best fixed / mixture growth bits per trial:",
      oracle_log_wealth/n/np.log(2), best_log_wealth/n/np.log(2),
      mixture_log_wealth/n/np.log(2))
print("Mixture regret to best fixed, bits:",
      (best_log_wealth-mixture_log_wealth)/np.log(2))

This is a controlled synthetic payoff experiment. The best fixed parameter uses the entire sequence and is a benchmark, while the mixture updates causally. Log wealth minus a best benchmark has the same algebra as negative-log-probability regret, with opposite orientation.

Modern view

Hellström, Durisi, Guedj and Raginsky’s published 2025 monograph [3] unifies information-theoretic and PAC-Bayesian generalization bounds. Its taxonomy separates expectation, probability and conditional-mutual-information formulations, making clear that a statistical guarantee needs a specified random object and evaluation criterion.

El Gamal’s 2026 proposed learning chapter [4] connects divergence identities with statistical learning through a textbook-oriented presentation. It is a draft supplement. We use its current perspective alongside the published second edition, without treating a proposed chapter as a completed new edition.

Our inference is that the same quantity can have several operational meanings only after the model is specified. Divergence can be a coding penalty, an unusual-type exponent, an evidence rate or a log-growth loss. The direction, logarithm base and limiting regime can differ among those uses.

Open problems in learned inference include reliable estimation of information in large parameter spaces, guarantees under dependent or shifted data, and bounds that remain informative for actual model selection procedures. A formal inequality can become vacuous when its information term is infinite or difficult to estimate.

For practical work, pair theoretical identities with independent finite experiments: exact toy probabilities to verify implementation, held-out likelihood to assess model fit, and decision-specific errors to evaluate use. No single entropy number replaces all three.

The final chapter combines multiple sources and receivers, develops stronger inequalities, and connects these foundations to modern learning and a small distributed-information capstone.

Key takeaways

  • Likelihood evidence, posterior odds and decision thresholds are distinct.
  • Testing reports both false-alarm and missed-alternative probabilities.
  • Type probabilities have divergence exponents with explicit argument orientation.
  • Hypothesis-testing exponents depend on the chosen error regime.
  • Sufficiency is relative to a model family and target parameter.
  • Fisher information gives local regular-estimation limits.
  • Maximum entropy depends on support and stated constraints.
  • Log-growth mismatch and universal mixture regret share coding identities.

Exercises

  1. Reverse the null and alternative in the binomial example. Find a threshold meeting a 5% false-alarm limit and compare the resulting missed-alternative error.
Hint

The rejection region becomes a low-count event.

  1. Derive the type-probability identity by grouping symbols with equal empirical counts. Explain why the divergence arguments have their displayed order.
Hint

Use counts times log source probability.

  1. Compare the Stein divergences in both directions for Bernoulli probabilities 0.3 and 0.6. Explain why the asymmetric results do not contradict a symmetric Chernoff exponent.
Hint

Fix the acceptance region before choosing a divergence.

  1. Reparameterize a Bernoulli probability by log odds. Derive its Fisher information and explain the changed error units.
Hint

Multiply by the squared probability derivative.

  1. Solve a maximum-entropy distribution on states 0, 1, 2, 3 with mean 1.2. Then add a second-moment constraint and describe the extra multiplier.
Hint

The second feature is squared state.

  1. In the fair contingent game, calculate expected growth at source probability 0.65 for allocations 0.5, 0.65 and 0.8. Verify the divergence loss and explain which strategy knows future outcomes.
Hint

None knows the future; the hindsight benchmark only fits a constant after observation.

Supplementary notes

1 supplementary note, each on a page of its own; the main text does not depend on them.

Open the notes →

References

  1. T. M. Cover and J. A. Thomas, Elements of Information Theory, 2nd ed., Wiley, 2006, Chapters 6, 11–12 and 16. Publisher book and contents.

  2. T. Weissman, “Method of Types,” Stanford EE276 Lecture 15 notes, 2024. Primary lecture notes.

  3. F. Hellström, G. Durisi, B. Guedj and M. Raginsky, “Generalization Bounds: Perspectives from Information Theory and PAC-Bayes,” Foundations and Trends in Machine Learning, 18(1), 1–223, 2025. Publisher metadata, author version.

  4. A. El Gamal, “Information Theory and Statistical Learning,” proposed textbook chapter, arXiv preprint, 2026. Draft and publication status.

  5. MIT OpenCourseWare, “Lecture 6: Efficient Estimators, Rao-Cramer Bound,” 14.381 Statistical Method in Economics, Fall 2018, pp. 3–6. Primary lecture notes.

Check your understanding

This tutorial has 6 exercises. Can you solve them without opening the hints?

Go to the exercises ↑

Next IT 08 · Networks, Information Inequalities and Learning · 19 min