Full search page →

PR 09 · Does averaging many unstable trees beat one tree?

AdvancedPR 09 · Ensembles, Validation and Cluster ValidityImage lab

Description

Grow real decision trees on 300 noisy two-moons points, bootstrap them, let them vote, and watch the test error fall and then flatten as trees are added.

Try this: Start with 50 trees at the fully grown depth 14 and drag M from 1 to 100: the vote falls from 14.60% to 11.10% over the first ten trees, then barely moves (10.25% at both 50 and 100). Switch the view to One member and drag M to see how ragged single bootstrap trees are, then back to the vote. Lower the depth to 2: the members now disagree on only 4.88% of the test points per pair, and the vote (10.70%) is no better than one tree (10.50%). Finally try depth 5: one tree already scores 9.30% against 10.15% for 50 bagged trees, a gap of less than one percentage point, about the noise of a 1000-point test set, so bagging does not replace choosing a sensible depth. Why does a vote of similar trees gain nothing?

Predict first

Fully grown trees (depth 14) are trained on 300 noisy moon points. Fifty more copies are trained on bootstrap resamples of the same points and vote on 1000 fresh test points. Compared with one fully grown tree trained on all 300 points, the vote will…

Hints

The chapter section that uses this lab:

Explanation

Answer the prediction first, or open the explanation when you are ready.

Show the explanation

Yes, at the lab’s seed: the single fully grown tree misclassifies 12.90% of the 1000 test points and the vote of 50 bagged trees 10.25%. Each member alone is worse than that tree (14.73% on average), because it sees only a resample of the points; the vote wins because the members do not make the same mistakes: a random pair of them disagrees on 13.70% of the test points. It cannot fall near zero. For 100 test points, more than half of the 50 trees are wrong, and adding trees cannot fix those: every tree learns from the same 300 noisy points, so they share the same bias and many of the same mistakes. This is the limit in the chapter’s formula, where the variance of the average of M predictors with pairwise correlation ρ is σ²[ρ + (1 − ρ)/M]: more trees only shrink the second term. Going from 50 to 100 trees changes nothing (10.25% both times), and even the best possible classifier would still make about 8.5% errors on this noisy data.

Lab · PR 09 · Does averaging many unstable trees beat one tree?

Predict first

Fully grown trees (depth 14) are trained on 300 noisy moon points. Fifty more copies are trained on bootstrap resamples of the same points and vote on 1000 fresh test points. Compared with one fully grown tree trained on all 300 points, the vote will…

Loading image…

Decision map of the chosen view over the 300 training points (circle = class 0, square = class 1; paler colour = the trees disagree; hollow points = missing from that member’s bootstrap sample)
Test error of the bagged vote for every M from 1 to 100 (1000 held-out points); the ring marks the current M

Try this: Start with 50 trees at the fully grown depth 14 and drag M from 1 to 100: the vote falls from 14.60% to 11.10% over the first ten trees, then barely moves (10.25% at both 50 and 100). Switch the view to One member and drag M to see how ragged single bootstrap trees are, then back to the vote. Lower the depth to 2: the members now disagree on only 4.88% of the test points per pair, and the vote (10.70%) is no better than one tree (10.50%). Finally try depth 5: one tree already scores 9.30% against 10.15% for 50 bagged trees, a gap of less than one percentage point, about the noise of a 1000-point test set, so bagging does not replace choosing a sensible depth. Why does a vote of similar trees gain nothing?

Data: the two-moons set from scikit-learn make_moons with Gaussian noise 0.3, generated once and stored in the page: 300 training points (150 per class, data seed 2) and a separate held-out test set of 1000 points (seed 1002). The data seed is the one of 30 random draws whose bagging gain is closest to the average gain, so this is a typical case, not a favourable one. Trees: CART with Gini impurity and axis-aligned splits at the midpoint between neighbouring values; the depth limit comes from the slider (14 grows every tree here until its leaves are pure). Bagging: member b is trained on 300 rows drawn with replacement from one fixed seeded stream (seed 20260618), so moving M only adds or removes trees and the curve does not change. The vote is a plain majority; for an even M an exact tie counts as half an error. Test error is measured on the 1000 held-out points; its standard error is about 1 percentage point at an error of 10%, so differences smaller than that between two different settings are weak evidence on their own. Disagreement: for every pair of the M members, the share of the 1000 test points on which the two predict different classes, averaged over all pairs (0 means identical members; the closer to 0, the more alike the members, which is a proxy for how correlated their mistakes are). Ties between equally good splits are broken by a fixed rule (feature 1 before feature 2, lower threshold first), while scikit-learn visits the features in random order: in the tests, trees without such a tie give predictions identical to DecisionTreeClassifier, and the vote error of the full 100-tree ensembles differs from the same ensembles built with scikit-learn trees by at most 0.7 percentage points. Best possible 8.5% is the error of the ideal classifier on the distribution that make_moons samples, computed by numerical integration of the two class densities; the lab does not use it. Real bagging also gives each tree out-of-bag points for free error estimates; this lab uses the separate test set instead.

Method and image sources

Open in the Playground →