Machine Learning
What is regularization?
Regularization prevents overfitting by penalizing model complexity.
What is regularisation?
With many features and few examples, a linear model learns overly large coefficients to explain the noise and overfits. Regularisation prevents this by adding a penalty on the size of the coefficients to the loss function:
# ordinary: min Σ(y - ŷ)²
# Ridge (L2): min Σ(y - ŷ)² + α·Σw²
# Lasso (L1): min Σ(y - ŷ)²/(2n) + α·Σ|w|
As α grows the coefficients shrink: the model becomes simpler and less flexible. α = 0 is ordinary linear regression.
Few examples, many features
In data with 60 examples and 20 features where only 3 features really matter, ordinary regression also finds non-zero coefficients for the other 17. The training score is high and the test score low:
m = LinearRegression().fit(X_tr, y_tr)
print(m.score(X_tr, y_tr), m.score(X_te, y_te)) # the gap is overfitting
The problem grows as the number of examples approaches the number of features; once features outnumber examples, ordinary regression no longer even has a unique solution.
Exercise See overfitting
Split the data with test_size=0.5, random_state=0. Train LinearRegression and Ridge(alpha=10); for each print the training and test R² with round(..., 3) on one line.
Show example solution
import numpy as np
from sklearn.preprocessing import StandardScaler
rng = np.random.default_rng(0)
X = rng.normal(size=(60, 20))
y = 3 * X[:, 0] - 2 * X[:, 1] + 1.5 * X[:, 2] + rng.normal(0, 1, 60)
Xs = StandardScaler().fit_transform(X)
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression, Ridge
X_tr, X_te, y_tr, y_te = train_test_split(Xs, y, test_size=0.5, random_state=0)
for m in [LinearRegression(), Ridge(alpha=10)]:
m.fit(X_tr, y_tr)
print(round(m.score(X_tr, y_tr), 3), round(m.score(X_te, y_te), 3))
Test yourself
-
Which problem does regularisation target?
- Imbalanced classes
- Slow training
- Overfitting
- Missing data
Show answer
Overfitting — By penalising large coefficients it makes it harder for the model to fit the noise.
-
Set the penalty strength.
m = Ridge(____=1.0)
Show answer
m = Ridge(alpha=1.0)
-
Match the concepts.
- α = 0
- Ordinary regression
- large α
- Small coefficients
- L2 penalty
- Ridge
- L1 penalty
- Lasso
Next lesson: L1 (Lasso)
