Machine Learning
L1 (Lasso)
L1 regularization performs feature selection by shrinking coefficients to zero.
Lasso (L1)
Lasso uses an absolute-value penalty. This penalty has an important property: it pushes the coefficients of weak features to exactly zero. That way the model both learns and selects features:
from sklearn.linear_model import Lasso
m = Lasso(alpha=0.1).fit(Xs, y)
print((m.coef_ != 0).sum()) # number of non-zero coefficients
print(np.where(m.coef_ != 0)[0]) # selected features
A sparse (mostly zero) solution produces interpretable, fast models.
Scale and instability
The penalty is applied equally to all coefficients; that's why features must be scaled first, otherwise a large-unit feature gets off "cheaply" with a small coefficient. Among correlated features, Lasso may also pick one arbitrarily:
pipe = make_pipeline(StandardScaler(), Lasso(alpha=0.1))
# x1 and x1 + small noise: Lasso usually keeps only one
With groups of correlated features, Lasso's choice may change from sample to sample; ElasticNet softens this problem.
Exercise Selection with Lasso
Train a Lasso(alpha=0.2) on the scaled data. Print the indices of the non-zero coefficients, then those coefficients with np.round(..., 2), on separate lines.
Show example solution
import numpy as np
from sklearn.preprocessing import StandardScaler
rng = np.random.default_rng(0)
X = rng.normal(size=(60, 20))
y = 3 * X[:, 0] - 2 * X[:, 1] + 1.5 * X[:, 2] + rng.normal(0, 1, 60)
Xs = StandardScaler().fit_transform(X)
from sklearn.linear_model import Lasso
m = Lasso(alpha=0.2).fit(Xs, y)
idx = np.where(m.coef_ != 0)[0]
print(idx)
print(np.round(m.coef_[idx], 2))
Test yourself
-
What property sets Lasso apart from Ridge?
- It uses no penalty
- It can set coefficients to exactly zero
- It needs no scaling
- It is faster
Show answer
It can set coefficients to exactly zero — The cornered geometry of the L1 penalty pushes the solution onto the axes, i.e. zero coefficients.
-
Count the non-zero coefficients.
print((m.coef_ ____ 0).sum())
Show answer
print((m.coef_ != 0).sum())
-
Match the properties.
- L1
- Absolute-value penalty
- sparse solution
- Most coefficients zero
- scaling
- Required for a fair penalty
- correlated features
- Risk of arbitrary choice
Next lesson: L2 (Ridge)
