What is boosting?
- Machine Learning course
- Intermediate
- Lesson 41/112
- 5 min
Boosting trains weak models sequentially to build a strong model.
What is boosting?
Bagging trains models in parallel and independently. Boosting trains models sequentially: each new model focuses on the examples the previous ones got wrong. The result is a strong model built from weak models (shallow trees).
# bagging: tree1, tree2, tree3 ... independent -> average (variance drops)
# boosting: tree1 -> tree2 fixes the errors -> tree3 ... (bias drops)The most successful methods on tabular data (XGBoost, LightGBM, scikit-learn's HistGradientBoosting) are from the boosting family.
The weak learner
The building block of boosting is a simple model slightly better than random; usually trees of depth 1-3. On its own a depth-1 tree (a stump) is weak, but the sequential sum of hundreds draws complex boundaries:
stump = DecisionTreeClassifier(max_depth=1).fit(X_train, y_train)
print(stump.score(X_test, y_test)) # one question: weak
ada = AdaBoostClassifier(n_estimators=200, random_state=0).fit(X_train, y_train)
print(ada.score(X_test, y_test)) # 200 stumps: strongUsing weak learners makes each step a small, controlled correction.
Practice Stump and boosting
Train a single DecisionTreeClassifier(max_depth=1) and an AdaBoostClassifier(n_estimators=200, random_state=0). Print the two test accuracies with round(..., 3) on separate lines.
Starter code
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import AdaBoostClassifier
# buraya yazSample solution
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import AdaBoostClassifier
stump = DecisionTreeClassifier(max_depth=1).fit(X_train, y_train)
ada = AdaBoostClassifier(n_estimators=200, random_state=0).fit(X_train, y_train)
print(round(stump.score(X_test, y_test), 3))
print(round(ada.score(X_test, y_test), 3))Expected output
0.888 0.944
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 Why can't the trees be trained in parallel in boosting?
- They can be
- Each tree depends on the previous ones' errors Correct answer
- The trees are very deep
- There isn't enough memory
Explanation: The sequential structure is the key feature separating boosting from bagging.
2 Build a one-question tree.
stump = DecisionTreeClassifier(____=1)stump = DecisionTreeClassifier(max_depth=1)Explanation: A depth-1 tree makes a single split.
3 Match the methods.
| Term | Meaning |
|---|---|
bagging |
Parallel, independent |
boosting |
Sequential, error-focused |
stump |
Depth-1 tree |
weak learner |
Slightly better than random |