Full search page →

When the support moves with the parameter

Optional · supplementary note 1 of 1 for IT 07 · Statistical Evidence, Maximum Entropy and Growth. Nothing in the main text depends on it; citation numbers refer to the reference list at the bottom of this page.

Navigate: this page and this track: Navigate: this page and this track

A derivative taken inside a density’s support can miss information carried by its moving boundary. The regular Cramér–Rao calculation then does not apply [5].

Let X1,…,XnX_1,\ldots,X_n be independent uniform observations on [0,θ][0,\theta], where θ>0\theta>0 is unknown. Inside the interval the one-sample score is −1/θ-1/\theta, whose expectation is not zero. The support also changes with θ\theta, so the regular score identities and information-additivity argument cannot be reused.

For the maximum M=max⁡iXiM=\max_i X_i, independence gives Pr⁡(M≤m)=(m/θ)n\Pr(M\leq m)=(m/\theta)^n for 0≤m≤θ0\leq m\leq\theta. Integrating this distribution yields

E[M]=nθn+1,T=n+1nM,Var⁡(T)=θ2n(n+2).\mathbb E[M]=\frac{n\theta}{n+1},\qquad T=\frac{n+1}{n}M,\qquad \operatorname{Var}(T)=\frac{\theta^2}{n(n+2)}.

Here nn is sample size, mm a possible maximum and TT an unbiased estimator of θ\theta. Its variance is smaller than the invalid bound θ2/n\theta^2/n obtained by mechanically substituting the squared one-sample score into the regular formula. The theorem is not contradicted: its assumptions fail.

Code · Python · 12 lines
import numpy as np

rng = np.random.default_rng(47)
theta, n = 3.0, 12
maximum = rng.uniform(0, theta, size=(50000, n)).max(axis=1)
estimate = (n+1)*maximum/n
exact_variance = theta**2/(n*(n+2))
assert abs(estimate.mean()-theta) < 0.01
assert abs(estimate.var()-exact_variance) < 0.005
assert exact_variance < theta**2/n
print("Estimator mean / target:", estimate.mean(), theta)
print("Estimator variance / exact:", estimate.var(), exact_variance)

The exact moments establish the claim; the simulation checks the implementation. This example is a reason to inspect parameter-dependent support before applying an information bound, not a general promise that estimation improves at this rate.

References

  1. T. M. Cover and J. A. Thomas, Elements of Information Theory, 2nd ed., Wiley, 2006, Chapters 6, 11–12 and 16. Publisher book and contents.

  2. T. Weissman, “Method of Types,” Stanford EE276 Lecture 15 notes, 2024. Primary lecture notes.

  3. F. Hellström, G. Durisi, B. Guedj and M. Raginsky, “Generalization Bounds: Perspectives from Information Theory and PAC-Bayes,” Foundations and Trends in Machine Learning, 18(1), 1–223, 2025. Publisher metadata, author version.

  4. A. El Gamal, “Information Theory and Statistical Learning,” proposed textbook chapter, arXiv preprint, 2026. Draft and publication status.

  5. MIT OpenCourseWare, “Lecture 6: Efficient Estimators, Rao-Cramer Bound,” 14.381 Statistical Method in Economics, Fall 2018, pp. 3–6. Primary lecture notes.