Machine Learning
alpha
The alpha parameter controls regularization strength.
Choosing alpha
alpha sets the balance between bias and variance: too small overfits, too large brings all coefficients near zero and underfits. The best value is found with CV on a logarithmic grid. scikit-learn offers ready-made classes for this:
from sklearn.linear_model import RidgeCV, LassoCV
r = RidgeCV(alphas=np.logspace(-3, 3, 13)).fit(Xs, y)
print(r.alpha_)
l = LassoCV(cv=5, random_state=0).fit(Xs, y)
print(l.alpha_)
RidgeCV uses a very fast leave-one-out computation by default.
The regularisation path
Seeing how the coefficients change with alpha makes the model easier to understand. On the Lasso path, as alpha grows the features drop to zero one by one; the last to drop are the most important features:
for a in [0.01, 0.1, 0.5, 1, 2]:
c = Lasso(alpha=a).fit(Xs, y).coef_
print(a, (c != 0).sum())
lasso_path computes the whole path at once, ready for plotting.
Exercise The Lasso path
Train Lasso for alpha = 0.01, 0.1, 0.5, 1, 2; on each line print alpha and the number of non-zero coefficients. Then print the alpha chosen by LassoCV(cv=5, random_state=0) with round(..., 3).
Show example solution
import numpy as np
from sklearn.preprocessing import StandardScaler
rng = np.random.default_rng(0)
X = rng.normal(size=(60, 20))
y = 3 * X[:, 0] - 2 * X[:, 1] + 1.5 * X[:, 2] + rng.normal(0, 1, 60)
Xs = StandardScaler().fit_transform(X)
from sklearn.linear_model import Lasso, LassoCV
for a in [0.01, 0.1, 0.5, 1, 2]:
print(a, (Lasso(alpha=a).fit(Xs, y).coef_ != 0).sum())
print(round(LassoCV(cv=5, random_state=0).fit(Xs, y).alpha_, 3))
Test yourself
-
What happens if alpha is chosen very large?
- The model overfits
- Nothing changes
- It errors
- The coefficients approach zero and the model underfits
Show answer
The coefficients approach zero and the model underfits — When the penalty dominates, the model stops fitting the data; predictions approach the mean.
-
Set up the Lasso that chooses alpha with CV.
l = ____(cv=5, random_state=0).fit(Xs, y)
Show answer
l = LassoCV(cv=5, random_state=0).fit(Xs, y)
-
Match the tools.
- RidgeCV
- Fast CV for Ridge
- LassoCV
- CV for Lasso
- alpha_
- Chosen penalty
- lasso_path
- Coefficient path
Next lesson: ElasticNet
