Machine Learning
Learning rate
Learning rate scales each tree's contribution.
Learning rate
learning_rate shrinks each tree's contribution. A small rate reaches the same success with more trees but generalises better; this is called shrinkage. The learning rate and the number of trees are tuned together:
for lr, n in [(1.0, 50), (0.1, 500)]:
m = GradientBoostingClassifier(learning_rate=lr, n_estimators=n, random_state=0)
print(lr, n, m.fit(X_train, y_train).score(X_test, y_test))
Unlike a random forest, in boosting overfitting does happen as the number of trees grows.
Early stopping
Instead of searching by hand for how many trees are enough, training can stop when the validation score stops improving. In HistGradientBoosting it is done automatically with an internal validation set:
m = HistGradientBoostingClassifier(max_iter=1000, early_stopping=True,
validation_fraction=0.1, n_iter_no_change=10,
random_state=0).fit(X_train, y_train)
print(m.n_iter_) # trees actually used
In classic GradientBoosting the same is done with the n_iter_no_change parameter.
Exercise Early stopping
Train a HistGradientBoostingClassifier(max_iter=1000, early_stopping=True, n_iter_no_change=10, random_state=0). Print the number of rounds used (n_iter_) and the test accuracy with round(..., 3) on separate lines.
Show example solution
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
from sklearn.ensemble import HistGradientBoostingClassifier
m = HistGradientBoostingClassifier(max_iter=1000, early_stopping=True, n_iter_no_change=10, random_state=0).fit(X_train, y_train)
print(m.n_iter_)
print(round(m.score(X_test, y_test), 3))
Test yourself
-
If the learning rate is halved, what is the general rule for the number of trees?
- It stays the same
- It drops to one
- It is roughly doubled
- It is halved
Show answer
It is roughly doubled — Each step gets smaller, so covering the same distance needs more steps.
-
Shrink each tree's contribution.
m = GradientBoostingClassifier(____=0.05, n_estimators=400)
Show answer
m = GradientBoostingClassifier(learning_rate=0.05, n_estimators=400)
-
Match the settings.
- learning_rate
- Step size
- n_estimators
- Number of trees
- n_iter_no_change
- Early-stopping patience
- n_iter_
- Rounds used
Next lesson: Advantages
