Machine Learning
Boosting
Boosting lets sequential models correct each other's errors.
Boosting among ensembles
Boosting combines not models of different kinds but the same weak learner sequentially. Bagging mainly reduces variance, boosting primarily bias; that's why they stand out in different situations:
- If the model overfits (high variance): bagging / random forest.
- If the model underfits (high bias): boosting.
from sklearn.ensemble import HistGradientBoostingClassifier
hgb = HistGradientBoostingClassifier(max_depth=3, learning_rate=0.1, random_state=0)
A boosting model can also be a member of a voting or stacking ensemble.
Observing bias and variance
Comparing bagging and boosting with the same weak learner makes the difference concrete. For stumps bagging gives almost no gain, while boosting makes a big difference:
stump = DecisionTreeClassifier(max_depth=1)
BaggingClassifier(stump, n_estimators=200) # close to the stump
AdaBoostClassifier(stump, n_estimators=200) # much better
Conversely, boosting with deep trees overfits fast, while bagging gives good results.
Exercise On stumps
Train a single stump, a BaggingClassifier with 200 stumps and an AdaBoostClassifier with 200 stumps (both random_state=0). Print the three test accuracies with round(..., 3) on one line.
Show example solution
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=2, stratify=y)
modeller = [("lr", make_pipeline(StandardScaler(), LogisticRegression())), ("knn", make_pipeline(StandardScaler(), KNeighborsClassifier())), ("agac", DecisionTreeClassifier(max_depth=4, random_state=0))]
from sklearn.ensemble import BaggingClassifier, AdaBoostClassifier
stump = DecisionTreeClassifier(max_depth=1)
m = [stump, BaggingClassifier(stump, n_estimators=200, random_state=0), AdaBoostClassifier(stump, n_estimators=200, random_state=0)]
print(*[round(x.fit(X_train, y_train).score(X_test, y_test), 3) for x in m])
Test yourself
-
If a simple model like a stump (depth 1) underfits, which ensemble suits better?
- A single model
- Boosting
- Bagging
- Neither
Show answer
Boosting — Bagging averages simple models and doesn't reduce bias; boosting corrects the errors step by step.
-
Build AdaBoost with stumps.
ada = AdaBoostClassifier(DecisionTreeClassifier(____=1), n_estimators=200)
Show answer
ada = AdaBoostClassifier(DecisionTreeClassifier(max_depth=1), n_estimators=200)
-
Match the situations.
- high variance
- Bagging
- high bias
- Boosting
- stump + bagging
- Little gain
- deep tree + boosting
- Overfitting risk
Next lesson: Stacking
