Bagging
- Machine Learning course
- Intermediate
- Lesson 38/112
- 5 min
Bagging trains each tree with random samples for diversity.
Bagging
Bagging (bootstrap aggregating) is the general method of replicating any model with bootstrap samples and averaging them. A random forest is bagging with "random features at each split" added:
from sklearn.ensemble import BaggingClassifier
bag = BaggingClassifier(DecisionTreeClassifier(), n_estimators=100, random_state=0)
bag.fit(X_train, y_train)Bagging helps most with unstable (high-variance) models: deep trees, KNN with a small k.
Out-of-bag (OOB) score
Each tree never sees ~37% of the examples. When each example is evaluated by the vote of the trees that haven't seen it, a success estimate is obtained without a separate validation set:
rf = RandomForestClassifier(n_estimators=200, oob_score=True, random_state=0)
rf.fit(X_train, y_train)
print(rf.oob_score_) # a free estimate close to CVThe OOB score is a fast alternative to CV on large datasets.
Practice OOB and test
Train a RandomForestClassifier(n_estimators=200, oob_score=True, random_state=0). Print oob_score_ and the test accuracy with round(..., 3) on one line. Then compute what share of the original examples a bootstrap sample contains with np.unique(rng.integers(0, n, n)).size / n and print it with round(..., 2) (n = 10000).
Starter code
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
rng = np.random.default_rng(0)
# buraya yazSample solution
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
rng = np.random.default_rng(0)
rf = RandomForestClassifier(n_estimators=200, oob_score=True, random_state=0).fit(X_train, y_train)
print(round(rf.oob_score_, 3), round(rf.score(X_test, y_test), 3))
n = 10000
print(round(np.unique(rng.integers(0, n, n)).size / n, 2))Expected output
0.965 0.944 0.64
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 Roughly what percentage of the original examples appear in a bootstrap sample?
- 63% Correct answer
- 100%
- 50%
- 37%
Explanation: The chance an example is never picked is (1-1/n)ⁿ ≈ e⁻¹ ≈ 0.37; the remaining 63% are picked at least once.
2 Have the out-of-bag score computed.
rf = RandomForestClassifier(n_estimators=200, ____=True)rf = RandomForestClassifier(n_estimators=200, oob_score=True)Explanation: oob_score=True fills the oob_score_ attribute after training.
3 Match the concepts.
| Term | Meaning |
|---|---|
bagging |
Bootstrap + averaging |
random forest |
Bagging + random features |
OOB |
Unseen examples |
oob_score_ |
Free validation |