Machine Learning
Bagging
Bagging trains each tree with random samples for diversity.
Bagging
Bagging (bootstrap aggregating) is the general method of replicating any model with bootstrap samples and averaging them. A random forest is bagging with "random features at each split" added:
from sklearn.ensemble import BaggingClassifier
bag = BaggingClassifier(DecisionTreeClassifier(), n_estimators=100, random_state=0)
bag.fit(X_train, y_train)
Bagging helps most with unstable (high-variance) models: deep trees, KNN with a small k.
Out-of-bag (OOB) score
Each tree never sees ~37% of the examples. When each example is evaluated by the vote of the trees that haven't seen it, a success estimate is obtained without a separate validation set:
rf = RandomForestClassifier(n_estimators=200, oob_score=True, random_state=0)
rf.fit(X_train, y_train)
print(rf.oob_score_) # a free estimate close to CV
The OOB score is a fast alternative to CV on large datasets.
Exercise OOB and test
Train a RandomForestClassifier(n_estimators=200, oob_score=True, random_state=0). Print oob_score_ and the test accuracy with round(..., 3) on one line. Then compute what share of the original examples a bootstrap sample contains with np.unique(rng.integers(0, n, n)).size / n and print it with round(..., 2) (n = 10000).
Show example solution
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
rng = np.random.default_rng(0)
rf = RandomForestClassifier(n_estimators=200, oob_score=True, random_state=0).fit(X_train, y_train)
print(round(rf.oob_score_, 3), round(rf.score(X_test, y_test), 3))
n = 10000
print(round(np.unique(rng.integers(0, n, n)).size / n, 2))
Test yourself
-
Roughly what percentage of the original examples appear in a bootstrap sample?
Show answer
63% — The chance an example is never picked is (1-1/n)ⁿ ≈ e⁻¹ ≈ 0.37; the remaining 63% are picked at least once.
-
Have the out-of-bag score computed.
rf = RandomForestClassifier(n_estimators=200, ____=True)
Show answer
rf = RandomForestClassifier(n_estimators=200, oob_score=True)
-
Match the concepts.
- bagging
- Bootstrap + averaging
- random forest
- Bagging + random features
- OOB
- Unseen examples
- oob_score_
- Free validation
Next lesson: Feature importance
