Machine Learning
L2 (Ridge)
L2 regularization shrinks coefficients but doesn't zero them.
Ridge (L2)
Ridge uses a squared penalty. It shrinks the coefficients towards zero but usually doesn't make them exactly zero. With correlated features it shares the weight between them; that's why it is more stable than Lasso:
from sklearn.linear_model import Ridge
for a in [0.01, 1, 100]:
m = Ridge(alpha=a).fit(Xs, y)
print(a, round(np.abs(m.coef_).sum(), 2), (m.coef_ == 0).sum()) # no zeros at all
Ridge has a closed-form solution: w = (XᵀX + αI)⁻¹Xᵀy; the αI term makes the solution possible even when XᵀX isn't invertible.
Ridge, Lasso and identical columns
The clearest way to see the difference is to copy a column:
X2 = np.column_stack([Xs[:, 0], Xs[:, 0]])
print(Ridge(alpha=1).fit(X2, y).coef_.round(2)) # [1.4 1.4] shares
print(Lasso(alpha=0.1).fit(X2, y).coef_.round(2)) # [2.7 0. ] picks one
Which behaviour is wanted depends on the question: Lasso for an interpretable sparse model, Ridge for stable predictions.
Exercise Identical columns
Train Ridge(alpha=1) and Lasso(alpha=0.1) with X2, made of two copies of the first column. Print the two coefficient arrays with np.round(..., 2) on separate lines.
Show example solution
import numpy as np
from sklearn.preprocessing import StandardScaler
rng = np.random.default_rng(0)
X = rng.normal(size=(60, 20))
y = 3 * X[:, 0] - 2 * X[:, 1] + 1.5 * X[:, 2] + rng.normal(0, 1, 60)
Xs = StandardScaler().fit_transform(X)
from sklearn.linear_model import Ridge, Lasso
X2 = np.column_stack([Xs[:, 0], Xs[:, 0]])
print(np.round(Ridge(alpha=1).fit(X2, y).coef_, 2))
print(np.round(Lasso(alpha=0.1).fit(X2, y).coef_, 2))
Test yourself
-
With two identical columns, how does Ridge distribute the coefficients?
- It shares them equally
- It zeroes one
- It zeroes both
- It errors
Show answer
It shares them equally — The squared penalty prefers equal coefficients when the total effect is the same.
-
Create the squared-penalty model.
m = ____(alpha=1.0)
Show answer
m = Ridge(alpha=1.0)
-
Match the behaviours.
- Ridge
- Shares the weight
- Lasso
- Picks one
- αI
- Ensures invertibility
- kare ceza
- Near zero, not zero
Next lesson: alpha
