What is regularization?
- Machine Learning course
- Intermediate
- Lesson 103/112
- 5 min
Regularization prevents overfitting by penalizing model complexity.
What is regularisation?
With many features and few examples, a linear model learns overly large coefficients to explain the noise and overfits. Regularisation prevents this by adding a penalty on the size of the coefficients to the loss function:
# ordinary: min Σ(y - ŷ)²
# Ridge (L2): min Σ(y - ŷ)² + α·Σw²
# Lasso (L1): min Σ(y - ŷ)²/(2n) + α·Σ|w|As α grows the coefficients shrink: the model becomes simpler and less flexible. α = 0 is ordinary linear regression.
Few examples, many features
In data with 60 examples and 20 features where only 3 features really matter, ordinary regression also finds non-zero coefficients for the other 17. The training score is high and the test score low:
m = LinearRegression().fit(X_tr, y_tr)
print(m.score(X_tr, y_tr), m.score(X_te, y_te)) # the gap is overfittingThe problem grows as the number of examples approaches the number of features; once features outnumber examples, ordinary regression no longer even has a unique solution.
Practice See overfitting
Split the data with test_size=0.5, random_state=0. Train LinearRegression and Ridge(alpha=10); for each print the training and test R² with round(..., 3) on one line.
Starter code
import numpy as np
from sklearn.preprocessing import StandardScaler
rng = np.random.default_rng(0)
X = rng.normal(size=(60, 20))
y = 3 * X[:, 0] - 2 * X[:, 1] + 1.5 * X[:, 2] + rng.normal(0, 1, 60)
Xs = StandardScaler().fit_transform(X)
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression, Ridge
# buraya yazSample solution
import numpy as np
from sklearn.preprocessing import StandardScaler
rng = np.random.default_rng(0)
X = rng.normal(size=(60, 20))
y = 3 * X[:, 0] - 2 * X[:, 1] + 1.5 * X[:, 2] + rng.normal(0, 1, 60)
Xs = StandardScaler().fit_transform(X)
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression, Ridge
X_tr, X_te, y_tr, y_te = train_test_split(Xs, y, test_size=0.5, random_state=0)
for m in [LinearRegression(), Ridge(alpha=10)]:
m.fit(X_tr, y_tr)
print(round(m.score(X_tr, y_tr), 3), round(m.score(X_te, y_te), 3))Expected output
0.981 0.555 0.875 0.731
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 Which problem does regularisation target?
- Imbalanced classes
- Slow training
- Overfitting Correct answer
- Missing data
Explanation: By penalising large coefficients it makes it harder for the model to fit the noise.
2 Set the penalty strength.
m = Ridge(____=1.0)m = Ridge(alpha=1.0)Explanation: In scikit-learn the penalty strength of Ridge and Lasso is alpha.
3 Match the concepts.
| Term | Meaning |
|---|---|
α = 0 |
Ordinary regression |
large α |
Small coefficients |
L2 penalty |
Ridge |
L1 penalty |
Lasso |