Advantages
- Machine Learning course
- Intermediate
- Lesson 45/112
- 5 min
Boosting provides high accuracy but is prone to overfitting.
Strengths of boosting
Gradient boosting often gives the highest score on tabular data. Its standout properties:
- It can lower both bias and variance (with the right tuning).
- It needs no scaling;
HistGradientBoostingsupports missing values and categorical features directly. - Different loss functions:
loss="absolute_error"for outlier-robust regression,"quantile"for prediction intervals.
Categorical features and NaN
HistGradientBoosting can split integer-coded categorical columns directly without one-hot, and learns a separate branch for missing values:
m = HistGradientBoostingClassifier(categorical_features=[0, 3], random_state=0)
m.fit(X_with_nan, y) # no SimpleImputer neededThis shortens the preprocessing chain and avoids the wide matrices one-hot produces for high-cardinality categories.
Practice Boosting with missing values
10% of the breast cancer data is set to np.nan (given). Train a HistGradientBoostingClassifier(random_state=0) directly and print the test accuracy with round(..., 3). Then print the number of NaNs in the training data.
Starter code
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
from sklearn.ensemble import HistGradientBoostingClassifier
rng = np.random.default_rng(0)
X_train = X_train.copy()
X_train[rng.random(X_train.shape) < 0.1] = np.nan
# buraya yazSample solution
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
from sklearn.ensemble import HistGradientBoostingClassifier
rng = np.random.default_rng(0)
X_train = X_train.copy()
X_train[rng.random(X_train.shape) < 0.1] = np.nan
m = HistGradientBoostingClassifier(random_state=0).fit(X_train, y_train)
print(round(m.score(X_test, y_test), 3))
print(np.isnan(X_train).sum())Expected output
0.93 1311
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 Which is a disadvantage of boosting compared with a random forest?
- It is always worse
- It needs more tuning and can overfit Correct answer
- Scaling is required
- It can't classify
Explanation: Learning rate, tree count and depth must be tuned together.
2 Choose an outlier-robust regression loss.
m = HistGradientBoostingRegressor(loss=____)m = HistGradientBoostingRegressor(loss="absolute_error")Explanation: Absolute error penalises large residuals less than squared error.
3 Match the features.
| Term | Meaning |
|---|---|
categorical_features |
Splits categories directly |
NaN support |
No imputer needed |
loss="quantile" |
Prediction interval |
loss="absolute_error" |
Robust to outliers |