PR 09 · Does averaging many unstable trees beat one tree?
AdvancedPR 09 · Ensembles, Validation and Cluster ValidityImage lab
Description
Grow real decision trees on 300 noisy two-moons points, bootstrap them, let them vote, and watch the test error fall and then flatten as trees are added.
Try this: Start with 50 trees at the fully grown depth 14 and drag M from 1 to 100: the vote falls from 14.60% to 11.10% over the first ten trees, then barely moves (10.25% at both 50 and 100). Switch the view to One member and drag M to see how ragged single bootstrap trees are, then back to the vote. Lower the depth to 2: the members now disagree on only 4.88% of the test points per pair, and the vote (10.70%) is no better than one tree (10.50%). Finally try depth 5: one tree already scores 9.30% against 10.15% for 50 bagged trees, a gap of less than one percentage point, about the noise of a 1000-point test set, so bagging does not replace choosing a sensible depth. Why does a vote of similar trees gain nothing?
Explanation
Answer the prediction first, or open the explanation when you are ready.
Show the explanation
Yes, at the lab’s seed: the single fully grown tree misclassifies 12.90% of the 1000 test points and the vote of 50 bagged trees 10.25%. Each member alone is worse than that tree (14.73% on average), because it sees only a resample of the points; the vote wins because the members do not make the same mistakes: a random pair of them disagrees on 13.70% of the test points. It cannot fall near zero. For 100 test points, more than half of the 50 trees are wrong, and adding trees cannot fix those: every tree learns from the same 300 noisy points, so they share the same bias and many of the same mistakes. This is the limit in the chapter’s formula, where the variance of the average of M predictors with pairwise correlation ρ is σ²[ρ + (1 − ρ)/M]: more trees only shrink the second term. Going from 50 to 100 trees changes nothing (10.25% both times), and even the best possible classifier would still make about 8.5% errors on this noisy data.
Lab · PR 09 · Does averaging many unstable trees beat one tree?
Loading image…
Try this: Start with 50 trees at the fully grown depth 14 and drag M from 1 to 100: the vote falls from 14.60% to 11.10% over the first ten trees, then barely moves (10.25% at both 50 and 100). Switch the view to One member and drag M to see how ragged single bootstrap trees are, then back to the vote. Lower the depth to 2: the members now disagree on only 4.88% of the test points per pair, and the vote (10.70%) is no better than one tree (10.50%). Finally try depth 5: one tree already scores 9.30% against 10.15% for 50 bagged trees, a gap of less than one percentage point, about the noise of a 1000-point test set, so bagging does not replace choosing a sensible depth. Why does a vote of similar trees gain nothing?
Data: the two-moons set from scikit-learn make_moons with Gaussian noise 0.3, generated once and stored in the page: 300 training points (150 per class, data seed 2) and a separate held-out test set of 1000 points (seed 1002). The data seed is the one of 30 random draws whose bagging gain is closest to the average gain, so this is a typical case, not a favourable one. Trees: CART with Gini impurity and axis-aligned splits at the midpoint between neighbouring values; the depth limit comes from the slider (14 grows every tree here until its leaves are pure). Bagging: member b is trained on 300 rows drawn with replacement from one fixed seeded stream (seed 20260618), so moving M only adds or removes trees and the curve does not change. The vote is a plain majority; for an even M an exact tie counts as half an error. Test error is measured on the 1000 held-out points; its standard error is about 1 percentage point at an error of 10%, so differences smaller than that between two different settings are weak evidence on their own. Disagreement: for every pair of the M members, the share of the 1000 test points on which the two predict different classes, averaged over all pairs (0 means identical members; the closer to 0, the more alike the members, which is a proxy for how correlated their mistakes are). Ties between equally good splits are broken by a fixed rule (feature 1 before feature 2, lower threshold first), while scikit-learn visits the features in random order: in the tests, trees without such a tie give predictions identical to DecisionTreeClassifier, and the vote error of the full 100-tree ensembles differs from the same ensembles built with scikit-learn trees by at most 0.7 percentage points. Best possible 8.5% is the error of the ideal classifier on the distribution that make_moons samples, computed by numerical integration of the two class densities; the lab does not use it. Real bagging also gives each tree out-of-bag points for free error estimates; this lab uses the separate test set instead.
Method and image sources
- L. Breiman, Bagging Predictors, Machine Learning, 24(2), 123-140, 1996 (bootstrap aggregating)
- L. Breiman, J. H. Friedman, R. A. Olshen and C. J. Stone, Classification and Regression Trees, Chapman and Hall/CRC (Wadsworth 1984), 2017 edition (CART, Gini impurity)
- F. Pedregosa et al., Scikit-learn: Machine Learning in Python, Journal of Machine Learning Research, 12, 2825-2830, 2011 (make_moons and the DecisionTreeClassifier used for the golden tests)
- scikit-learn documentation, sklearn.datasets.make_moons (the generated two-moons data, no download)