Bagging
- Machine Learning course
- Intermediate
- Lesson 99/112
- 5 min
Bagging averages predictions from parallel models.
Stabilising an unstable model with bagging
Bagging can be applied to any unstable model. High-variance models like a deep decision tree or KNN with a small k improve clearly with the average of bootstrap copies; stable models (logistic regression) gain little:
from sklearn.ensemble import BaggingClassifier
bag = BaggingClassifier(DecisionTreeClassifier(), n_estimators=100,
max_samples=0.8, max_features=0.8, random_state=0)max_features < 1 gives each copy a subset of the features and increases diversity.
Bagging and random forest
Bagging with trees is very close to a random forest; the difference is that a random forest picks random features at every split. BaggingClassifier(max_features=...) picks features once per tree:
# bagging(max_features=0.5): each tree sees half the features
# random forest(max_features="sqrt"): a new random subset at every splitSplit-level randomness makes the trees more diverse; that's why a random forest is usually preferred for trees.
Practice Bagging for KNN
Compare KNeighborsClassifier(n_neighbors=1) (scaled) with its 50-copy BaggingClassifier(..., max_samples=0.7, random_state=0) version using 5-fold CV. Print both means with round(..., 3) on one line.
Starter code
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=2, stratify=y)
modeller = [("lr", make_pipeline(StandardScaler(), LogisticRegression())), ("knn", make_pipeline(StandardScaler(), KNeighborsClassifier())), ("agac", DecisionTreeClassifier(max_depth=4, random_state=0))]
from sklearn.ensemble import BaggingClassifier
from sklearn.model_selection import cross_val_score
# buraya yazSample solution
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=2, stratify=y)
modeller = [("lr", make_pipeline(StandardScaler(), LogisticRegression())), ("knn", make_pipeline(StandardScaler(), KNeighborsClassifier())), ("agac", DecisionTreeClassifier(max_depth=4, random_state=0))]
from sklearn.ensemble import BaggingClassifier
from sklearn.model_selection import cross_val_score
tek = make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=1))
bag = BaggingClassifier(tek, n_estimators=50, max_samples=0.7, random_state=0)
print(round(cross_val_score(tek, X, y, cv=5).mean(), 3), round(cross_val_score(bag, X, y, cv=5).mean(), 3))Expected output
0.954 0.961
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 For which model does bagging give the least gain?
- KNN with k=1
- An unpruned tree
- Logistic regression Correct answer
- A deep decision tree
Explanation: Linear models change little when the data changes; averaging has little effect.
2 Give each copy 80% of the data.
bag = BaggingClassifier(model, n_estimators=100, ____=0.8)bag = BaggingClassifier(model, n_estimators=100, max_samples=0.8)Explanation: max_samples is each copy's share of the examples.
3 Match the parameters.
| Term | Meaning |
|---|---|
max_samples |
Examples per copy |
max_features |
Features per copy |
bootstrap |
Drawing with replacement |
random forest |
Per-split randomness |