Machine Learning
What is boosting?
Boosting trains weak models sequentially to build a strong model.
What is boosting?
Bagging trains models in parallel and independently. Boosting trains models sequentially: each new model focuses on the examples the previous ones got wrong. The result is a strong model built from weak models (shallow trees).
# bagging: tree1, tree2, tree3 ... independent -> average (variance drops)
# boosting: tree1 -> tree2 fixes the errors -> tree3 ... (bias drops)
The most successful methods on tabular data (XGBoost, LightGBM, scikit-learn's HistGradientBoosting) are from the boosting family.
The weak learner
The building block of boosting is a simple model slightly better than random; usually trees of depth 1-3. On its own a depth-1 tree (a stump) is weak, but the sequential sum of hundreds draws complex boundaries:
stump = DecisionTreeClassifier(max_depth=1).fit(X_train, y_train)
print(stump.score(X_test, y_test)) # one question: weak
ada = AdaBoostClassifier(n_estimators=200, random_state=0).fit(X_train, y_train)
print(ada.score(X_test, y_test)) # 200 stumps: strong
Using weak learners makes each step a small, controlled correction.
Exercise Stump and boosting
Train a single DecisionTreeClassifier(max_depth=1) and an AdaBoostClassifier(n_estimators=200, random_state=0). Print the two test accuracies with round(..., 3) on separate lines.
Show example solution
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import AdaBoostClassifier
stump = DecisionTreeClassifier(max_depth=1).fit(X_train, y_train)
ada = AdaBoostClassifier(n_estimators=200, random_state=0).fit(X_train, y_train)
print(round(stump.score(X_test, y_test), 3))
print(round(ada.score(X_test, y_test), 3))
Test yourself
-
Why can't the trees be trained in parallel in boosting?
- They can be
- Each tree depends on the previous ones' errors
- The trees are very deep
- There isn't enough memory
Show answer
Each tree depends on the previous ones' errors — The sequential structure is the key feature separating boosting from bagging.
-
Build a one-question tree.
stump = DecisionTreeClassifier(____=1)
Show answer
stump = DecisionTreeClassifier(max_depth=1)
-
Match the methods.
- bagging
- Parallel, independent
- boosting
- Sequential, error-focused
- stump
- Depth-1 tree
- weak learner
- Slightly better than random
Next lesson: AdaBoost
