Learning rate
- Machine Learning course
- Intermediate
- Lesson 44/112
- 5 min
Learning rate scales each tree's contribution.
Learning rate
learning_rate shrinks each tree's contribution. A small rate reaches the same success with more trees but generalises better; this is called shrinkage. The learning rate and the number of trees are tuned together:
for lr, n in [(1.0, 50), (0.1, 500)]:
m = GradientBoostingClassifier(learning_rate=lr, n_estimators=n, random_state=0)
print(lr, n, m.fit(X_train, y_train).score(X_test, y_test))Unlike a random forest, in boosting overfitting does happen as the number of trees grows.
Early stopping
Instead of searching by hand for how many trees are enough, training can stop when the validation score stops improving. In HistGradientBoosting it is done automatically with an internal validation set:
m = HistGradientBoostingClassifier(max_iter=1000, early_stopping=True,
validation_fraction=0.1, n_iter_no_change=10,
random_state=0).fit(X_train, y_train)
print(m.n_iter_) # trees actually usedIn classic GradientBoosting the same is done with the n_iter_no_change parameter.
Practice Early stopping
Train a HistGradientBoostingClassifier(max_iter=1000, early_stopping=True, n_iter_no_change=10, random_state=0). Print the number of rounds used (n_iter_) and the test accuracy with round(..., 3) on separate lines.
Starter code
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
from sklearn.ensemble import HistGradientBoostingClassifier
# buraya yazSample solution
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
from sklearn.ensemble import HistGradientBoostingClassifier
m = HistGradientBoostingClassifier(max_iter=1000, early_stopping=True, n_iter_no_change=10, random_state=0).fit(X_train, y_train)
print(m.n_iter_)
print(round(m.score(X_test, y_test), 3))Expected output
61 0.937
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 If the learning rate is halved, what is the general rule for the number of trees?
- It stays the same
- It drops to one
- It is roughly doubled Correct answer
- It is halved
Explanation: Each step gets smaller, so covering the same distance needs more steps.
2 Shrink each tree's contribution.
m = GradientBoostingClassifier(____=0.05, n_estimators=400)m = GradientBoostingClassifier(learning_rate=0.05, n_estimators=400)Explanation: In scikit-learn the parameter is named learning_rate.
3 Match the settings.
| Term | Meaning |
|---|---|
learning_rate |
Step size |
n_estimators |
Number of trees |
n_iter_no_change |
Early-stopping patience |
n_iter_ |
Rounds used |