AdaBoost

AdaBoost increases the weight of misclassified samples for the next model.

AdaBoost

AdaBoost increases the weight of misclassified examples in every round; the next weak learner pays more attention to these "hard" examples. Each learner's vote is weighted by how accurate it is:

ada = AdaBoostClassifier(
    estimator=DecisionTreeClassifier(max_depth=1),
    n_estimators=200, learning_rate=0.5, random_state=0)
ada.fit(X_train, y_train)
print(ada.estimator_weights_[:5])   # learner vote weights

Because of the reweighting, AdaBoost is sensitive to label noise and outliers: a mislabelled example gets more and more attention.

Error and vote weight

In binary AdaBoost, a learner with error ε has the vote weight α = ½·ln((1-ε)/ε). If ε = 0.5 (random), α = 0: the vote has no effect. As the error drops, the vote weight grows:

import numpy as np
for e in [0.45, 0.3, 0.1]:
    print(e, round(0.5 * np.log((1 - e) / e), 3))

That's why it is enough for the weak learner to be slightly better than random; its contribution is adjusted accordingly.

Exercise Vote weights

For the error rates [0.45, 0.3, 0.1], compute the vote weight with 0.5 * np.log((1 - e) / e) and print it with np.round(..., 3).

Show example solution

import numpy as np

e = np.array([0.45, 0.3, 0.1])

alfa = 0.5 * np.log((1 - e) / e)
print(np.round(alfa, 3))

Test yourself

In AdaBoost, what happens to a misclassified example in the next round?
- Its weight increases
- It is removed from the data
- Its weight decreases
- Its label is changed
Show answer

Its weight increases — The next learner is pushed harder to classify this example correctly.

Give a stump as the weak learner.

ada = AdaBoostClassifier(____=DecisionTreeClassifier(max_depth=1))

Show answer

ada = AdaBoostClassifier(estimator=DecisionTreeClassifier(max_depth=1))

Match the concepts.

sample weight

Grows on hard examples
α

Learner's vote weight
ε = 0.5

Vote weight 0
label noise

AdaBoost's weakness

Next lesson: Gradient boosting

We Are Enhancing Your Experience

Cookie Preferences

Strictly Necessary Cookies

Analytical Cookies