Boosting
- Machine Learning course
- Intermediate
- Lesson 100/112
- 5 min
Boosting lets sequential models correct each other's errors.
Boosting among ensembles
Boosting combines not models of different kinds but the same weak learner sequentially. Bagging mainly reduces variance, boosting primarily bias; that's why they stand out in different situations:
- If the model overfits (high variance): bagging / random forest.
- If the model underfits (high bias): boosting.
from sklearn.ensemble import HistGradientBoostingClassifier
hgb = HistGradientBoostingClassifier(max_depth=3, learning_rate=0.1, random_state=0)A boosting model can also be a member of a voting or stacking ensemble.
Observing bias and variance
Comparing bagging and boosting with the same weak learner makes the difference concrete. For stumps bagging gives almost no gain, while boosting makes a big difference:
stump = DecisionTreeClassifier(max_depth=1)
BaggingClassifier(stump, n_estimators=200) # close to the stump
AdaBoostClassifier(stump, n_estimators=200) # much betterConversely, boosting with deep trees overfits fast, while bagging gives good results.
Practice On stumps
Train a single stump, a BaggingClassifier with 200 stumps and an AdaBoostClassifier with 200 stumps (both random_state=0). Print the three test accuracies with round(..., 3) on one line.
Starter code
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=2, stratify=y)
modeller = [("lr", make_pipeline(StandardScaler(), LogisticRegression())), ("knn", make_pipeline(StandardScaler(), KNeighborsClassifier())), ("agac", DecisionTreeClassifier(max_depth=4, random_state=0))]
from sklearn.ensemble import BaggingClassifier, AdaBoostClassifier
# buraya yazSample solution
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=2, stratify=y)
modeller = [("lr", make_pipeline(StandardScaler(), LogisticRegression())), ("knn", make_pipeline(StandardScaler(), KNeighborsClassifier())), ("agac", DecisionTreeClassifier(max_depth=4, random_state=0))]
from sklearn.ensemble import BaggingClassifier, AdaBoostClassifier
stump = DecisionTreeClassifier(max_depth=1)
m = [stump, BaggingClassifier(stump, n_estimators=200, random_state=0), AdaBoostClassifier(stump, n_estimators=200, random_state=0)]
print(*[round(x.fit(X_train, y_train).score(X_test, y_test), 3) for x in m])Expected output
0.912 0.912 0.965
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 If a simple model like a stump (depth 1) underfits, which ensemble suits better?
- A single model
- Boosting Correct answer
- Bagging
- Neither
Explanation: Bagging averages simple models and doesn't reduce bias; boosting corrects the errors step by step.
2 Build AdaBoost with stumps.
ada = AdaBoostClassifier(DecisionTreeClassifier(____=1), n_estimators=200)ada = AdaBoostClassifier(DecisionTreeClassifier(max_depth=1), n_estimators=200)Explanation: max_depth=1 is a single-split tree.
3 Match the situations.
| Term | Meaning |
|---|---|
high variance |
Bagging |
high bias |
Boosting |
stump + bagging |
Little gain |
deep tree + boosting |
Overfitting risk |