Machine Learning
Bagging
Bagging averages predictions from parallel models.
Stabilising an unstable model with bagging
Bagging can be applied to any unstable model. High-variance models like a deep decision tree or KNN with a small k improve clearly with the average of bootstrap copies; stable models (logistic regression) gain little:
from sklearn.ensemble import BaggingClassifier
bag = BaggingClassifier(DecisionTreeClassifier(), n_estimators=100,
max_samples=0.8, max_features=0.8, random_state=0)
max_features < 1 gives each copy a subset of the features and increases diversity.
Bagging and random forest
Bagging with trees is very close to a random forest; the difference is that a random forest picks random features at every split. BaggingClassifier(max_features=...) picks features once per tree:
# bagging(max_features=0.5): each tree sees half the features
# random forest(max_features="sqrt"): a new random subset at every split
Split-level randomness makes the trees more diverse; that's why a random forest is usually preferred for trees.
Exercise Bagging for KNN
Compare KNeighborsClassifier(n_neighbors=1) (scaled) with its 50-copy BaggingClassifier(..., max_samples=0.7, random_state=0) version using 5-fold CV. Print both means with round(..., 3) on one line.
Show example solution
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=2, stratify=y)
modeller = [("lr", make_pipeline(StandardScaler(), LogisticRegression())), ("knn", make_pipeline(StandardScaler(), KNeighborsClassifier())), ("agac", DecisionTreeClassifier(max_depth=4, random_state=0))]
from sklearn.ensemble import BaggingClassifier
from sklearn.model_selection import cross_val_score
tek = make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=1))
bag = BaggingClassifier(tek, n_estimators=50, max_samples=0.7, random_state=0)
print(round(cross_val_score(tek, X, y, cv=5).mean(), 3), round(cross_val_score(bag, X, y, cv=5).mean(), 3))
Test yourself
-
For which model does bagging give the least gain?
- KNN with k=1
- An unpruned tree
- Logistic regression
- A deep decision tree
Show answer
Logistic regression — Linear models change little when the data changes; averaging has little effect.
-
Give each copy 80% of the data.
bag = BaggingClassifier(model, n_estimators=100, ____=0.8)
Show answer
bag = BaggingClassifier(model, n_estimators=100, max_samples=0.8)
-
Match the parameters.
- max_samples
- Examples per copy
- max_features
- Features per copy
- bootstrap
- Drawing with replacement
- random forest
- Per-split randomness
Next lesson: Boosting
