Gradient boosting
- Machine Learning course
- Intermediate
- Lesson 43/112
- 5 min
Gradient boosting improves the model by reducing errors with gradients.
Gradient boosting
Gradient boosting fits each new tree to the residuals of the current total model (y - ŷ in regression). So every tree learns "how wrong the prediction so far is" and is added to the prediction with a small step:
p = np.full(len(y), y.mean()) # start: the mean
for _ in range(100):
kalinti = y - p
agac = DecisionTreeRegressor(max_depth=2).fit(X, kalinti)
p += 0.1 * agac.predict(X) # small stepThe name comes from the gradient of the loss function: the negative gradient of squared error is exactly the residual.
Implementations in scikit-learn
There are two versions:
GradientBoostingClassifier/Regressor: classic, for small to medium data.HistGradientBoostingClassifier/Regressor: bins the features; much faster on tens of thousands of examples and supports missing values directly.
from sklearn.ensemble import HistGradientBoostingClassifier
hgb = HistGradientBoostingClassifier(random_state=0).fit(X_train, y_train) # NaN allowedOn large tabular data the first model to try is usually HistGradientBoosting.
Practice Gradient boosting by hand
Write a gradient boosting loop by hand with an initial prediction of y.mean(): 50 rounds, each fitting a DecisionTreeRegressor(max_depth=2) to the residuals and adding it to the prediction times 0.1. Print the MSE at the start and at the end with round(..., 2) on separate lines.
Starter code
import numpy as np
from sklearn.tree import DecisionTreeRegressor
rng = np.random.default_rng(0)
X = rng.uniform(0, 6, size=(200, 1))
y = np.sin(X[:, 0]) + rng.normal(0, 0.1, 200)
# buraya yazSample solution
import numpy as np
from sklearn.tree import DecisionTreeRegressor
rng = np.random.default_rng(0)
X = rng.uniform(0, 6, size=(200, 1))
y = np.sin(X[:, 0]) + rng.normal(0, 0.1, 200)
p = np.full(len(y), y.mean())
print(round(((y - p) ** 2).mean(), 2))
for _ in range(50):
agac = DecisionTreeRegressor(max_depth=2).fit(X, y - p)
p += 0.1 * agac.predict(X)
print(round(((y - p) ** 2).mean(), 2))Expected output
0.5 0.01
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 In gradient boosting regression, what does each new tree target?
- The original y
- Random labels
- The previous tree's prediction
- The current model's residuals Correct answer
Explanation: Residual = y - ŷ; the tree learns to shrink that difference.
2 Fit the tree to the residuals.
agac = DecisionTreeRegressor(max_depth=2).fit(X, ____)agac = DecisionTreeRegressor(max_depth=2).fit(X, y - p)Explanation: The target is the difference between the truth and the current prediction.
3 Match the concepts.
| Term | Meaning |
|---|---|
residual |
y - ŷ |
initial prediction |
The mean |
HistGradientBoosting |
Fast, NaN-capable |
negative gradient |
Residual for squared error |