When the support moves with the parameter
Optional · supplementary note 1 of 1 for IT 07 · Statistical Evidence, Maximum Entropy and Growth. Nothing in the main text depends on it; citation numbers refer to the reference list at the bottom of this page.
Navigate: this page and this track: Navigate: this page and this track
A derivative taken inside a density’s support can miss information carried by its moving boundary. The regular Cramér–Rao calculation then does not apply [5].
Let be independent uniform observations on , where is unknown. Inside the interval the one-sample score is , whose expectation is not zero. The support also changes with , so the regular score identities and information-additivity argument cannot be reused.
For the maximum , independence gives for . Integrating this distribution yields
Here is sample size, a possible maximum and an unbiased estimator of . Its variance is smaller than the invalid bound obtained by mechanically substituting the squared one-sample score into the regular formula. The theorem is not contradicted: its assumptions fail.
Code · Python · 12 lines
import numpy as np
rng = np.random.default_rng(47)
theta, n = 3.0, 12
maximum = rng.uniform(0, theta, size=(50000, n)).max(axis=1)
estimate = (n+1)*maximum/n
exact_variance = theta**2/(n*(n+2))
assert abs(estimate.mean()-theta) < 0.01
assert abs(estimate.var()-exact_variance) < 0.005
assert exact_variance < theta**2/n
print("Estimator mean / target:", estimate.mean(), theta)
print("Estimator variance / exact:", estimate.var(), exact_variance)The exact moments establish the claim; the simulation checks the implementation. This example is a reason to inspect parameter-dependent support before applying an information bound, not a general promise that estimation improves at this rate.
References
-
T. M. Cover and J. A. Thomas, Elements of Information Theory, 2nd ed., Wiley, 2006, Chapters 6, 11–12 and 16. Publisher book and contents.
-
T. Weissman, “Method of Types,” Stanford EE276 Lecture 15 notes, 2024. Primary lecture notes.
-
F. Hellström, G. Durisi, B. Guedj and M. Raginsky, “Generalization Bounds: Perspectives from Information Theory and PAC-Bayes,” Foundations and Trends in Machine Learning, 18(1), 1–223, 2025. Publisher metadata, author version.
-
A. El Gamal, “Information Theory and Statistical Learning,” proposed textbook chapter, arXiv preprint, 2026. Draft and publication status.
-
MIT OpenCourseWare, “Lecture 6: Efficient Estimators, Rao-Cramer Bound,” 14.381 Statistical Method in Economics, Fall 2018, pp. 3–6. Primary lecture notes.