Machine Learning
Gradient boosting
Gradient boosting improves the model by reducing errors with gradients.
Gradient boosting
Gradient boosting fits each new tree to the residuals of the current total model (y - ŷ in regression). So every tree learns "how wrong the prediction so far is" and is added to the prediction with a small step:
p = np.full(len(y), y.mean()) # start: the mean
for _ in range(100):
kalinti = y - p
agac = DecisionTreeRegressor(max_depth=2).fit(X, kalinti)
p += 0.1 * agac.predict(X) # small step
The name comes from the gradient of the loss function: the negative gradient of squared error is exactly the residual.
Implementations in scikit-learn
There are two versions:
GradientBoostingClassifier/Regressor: classic, for small to medium data.HistGradientBoostingClassifier/Regressor: bins the features; much faster on tens of thousands of examples and supports missing values directly.
from sklearn.ensemble import HistGradientBoostingClassifier
hgb = HistGradientBoostingClassifier(random_state=0).fit(X_train, y_train) # NaN allowed
On large tabular data the first model to try is usually HistGradientBoosting.
Exercise Gradient boosting by hand
Write a gradient boosting loop by hand with an initial prediction of y.mean(): 50 rounds, each fitting a DecisionTreeRegressor(max_depth=2) to the residuals and adding it to the prediction times 0.1. Print the MSE at the start and at the end with round(..., 2) on separate lines.
Show example solution
import numpy as np
from sklearn.tree import DecisionTreeRegressor
rng = np.random.default_rng(0)
X = rng.uniform(0, 6, size=(200, 1))
y = np.sin(X[:, 0]) + rng.normal(0, 0.1, 200)
p = np.full(len(y), y.mean())
print(round(((y - p) ** 2).mean(), 2))
for _ in range(50):
agac = DecisionTreeRegressor(max_depth=2).fit(X, y - p)
p += 0.1 * agac.predict(X)
print(round(((y - p) ** 2).mean(), 2))
Test yourself
-
In gradient boosting regression, what does each new tree target?
- The original y
- Random labels
- The previous tree's prediction
- The current model's residuals
Show answer
The current model's residuals — Residual = y - ŷ; the tree learns to shrink that difference.
-
Fit the tree to the residuals.
agac = DecisionTreeRegressor(max_depth=2).fit(X, ____)
Show answer
agac = DecisionTreeRegressor(max_depth=2).fit(X, y - p)
-
Match the concepts.
- residual
- y - ŷ
- initial prediction
- The mean
- HistGradientBoosting
- Fast, NaN-capable
- negative gradient
- Residual for squared error
Next lesson: Learning rate
