Summary - Boosting

In this group you learned AdaBoost and gradient boosting.

Summary: boosting

In this group:

Boosting trains weak learners sequentially; each focuses on the previous ones' errors.
AdaBoost changes example weights and weights votes with α = ½ln((1-ε)/ε); it is noise-sensitive.
Gradient boosting fits trees to residuals (the negative gradient).
learning_rate and the tree count are tuned together; early stopping prevents overfitting.
HistGradientBoosting: fast, with NaN and category support.

Exercise Three ensembles

Train RandomForestClassifier(random_state=0), GradientBoostingClassifier(random_state=0) and HistGradientBoostingClassifier(random_state=0) on the same split. On each line print the class name and the test accuracy with round(..., 3).

Show example solution

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier, HistGradientBoostingClassifier

for m in [RandomForestClassifier(random_state=0), GradientBoostingClassifier(random_state=0), HistGradientBoostingClassifier(random_state=0)]:
    m.fit(X_train, y_train)
    print(type(m).__name__, round(m.score(X_test, y_test), 3))

Test yourself

In boosting, if the validation score starts dropping after tree 300, what should be done?
- Early stopping or fewer trees
- Adding more trees
- Raising the learning rate
- Nothing
Show answer

Early stopping or fewer trees — Too many trees cause overfitting in boosting.

Create the classic gradient boosting classifier.

m = ____(n_estimators=200, learning_rate=0.1, max_depth=3)

Show answer

m = GradientBoostingClassifier(n_estimators=200, learning_rate=0.1, max_depth=3)

Match the methods.

AdaBoost

Example weights
Gradient boosting

Fitting residuals
HistGradientBoosting

Binned, fast
Random forest

Parallel averaging

Next lesson: What is clustering?

We Are Enhancing Your Experience

Cookie Preferences

Strictly Necessary Cookies

Analytical Cookies