Summary - Boosting
- Machine Learning course
- Intermediate
- Lesson 46/112
- 6 min
In this group you learned AdaBoost and gradient boosting.
Summary: boosting
In this group:
- Boosting trains weak learners sequentially; each focuses on the previous ones' errors.
- AdaBoost changes example weights and weights votes with
α = ½ln((1-ε)/ε); it is noise-sensitive. - Gradient boosting fits trees to residuals (the negative gradient).
learning_rateand the tree count are tuned together; early stopping prevents overfitting.HistGradientBoosting: fast, with NaN and category support.
Practice Three ensembles
Train RandomForestClassifier(random_state=0), GradientBoostingClassifier(random_state=0) and HistGradientBoostingClassifier(random_state=0) on the same split. On each line print the class name and the test accuracy with round(..., 3).
Starter code
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier, HistGradientBoostingClassifier
# buraya yazSample solution
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier, HistGradientBoostingClassifier
for m in [RandomForestClassifier(random_state=0), GradientBoostingClassifier(random_state=0), HistGradientBoostingClassifier(random_state=0)]:
m.fit(X_train, y_train)
print(type(m).__name__, round(m.score(X_test, y_test), 3))Expected output
RandomForestClassifier 0.944 GradientBoostingClassifier 0.958 HistGradientBoostingClassifier 0.937
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 In boosting, if the validation score starts dropping after tree 300, what should be done?
- Early stopping or fewer trees Correct answer
- Adding more trees
- Raising the learning rate
- Nothing
Explanation: Too many trees cause overfitting in boosting.
2 Create the classic gradient boosting classifier.
Question
m = ____(n_estimators=200, learning_rate=0.1, max_depth=3)
Answer
m = GradientBoostingClassifier(n_estimators=200, learning_rate=0.1, max_depth=3)Explanation: max_depth here is the depth of each tree.
3 Match the methods.
| Term | Meaning |
|---|---|
AdaBoost |
Example weights |
Gradient boosting |
Fitting residuals |
HistGradientBoosting |
Binned, fast |
Random forest |
Parallel averaging |