alpha
- Machine Learning course
- Intermediate
- Lesson 106/112
- 5 min
The alpha parameter controls regularization strength.
Choosing alpha
alpha sets the balance between bias and variance: too small overfits, too large brings all coefficients near zero and underfits. The best value is found with CV on a logarithmic grid. scikit-learn offers ready-made classes for this:
from sklearn.linear_model import RidgeCV, LassoCV
r = RidgeCV(alphas=np.logspace(-3, 3, 13)).fit(Xs, y)
print(r.alpha_)
l = LassoCV(cv=5, random_state=0).fit(Xs, y)
print(l.alpha_)RidgeCV uses a very fast leave-one-out computation by default.
The regularisation path
Seeing how the coefficients change with alpha makes the model easier to understand. On the Lasso path, as alpha grows the features drop to zero one by one; the last to drop are the most important features:
for a in [0.01, 0.1, 0.5, 1, 2]:
c = Lasso(alpha=a).fit(Xs, y).coef_
print(a, (c != 0).sum())lasso_path computes the whole path at once, ready for plotting.
Practice The Lasso path
Train Lasso for alpha = 0.01, 0.1, 0.5, 1, 2; on each line print alpha and the number of non-zero coefficients. Then print the alpha chosen by LassoCV(cv=5, random_state=0) with round(..., 3).
Starter code
import numpy as np
from sklearn.preprocessing import StandardScaler
rng = np.random.default_rng(0)
X = rng.normal(size=(60, 20))
y = 3 * X[:, 0] - 2 * X[:, 1] + 1.5 * X[:, 2] + rng.normal(0, 1, 60)
Xs = StandardScaler().fit_transform(X)
from sklearn.linear_model import Lasso, LassoCV
# buraya yazSample solution
import numpy as np
from sklearn.preprocessing import StandardScaler
rng = np.random.default_rng(0)
X = rng.normal(size=(60, 20))
y = 3 * X[:, 0] - 2 * X[:, 1] + 1.5 * X[:, 2] + rng.normal(0, 1, 60)
Xs = StandardScaler().fit_transform(X)
from sklearn.linear_model import Lasso, LassoCV
for a in [0.01, 0.1, 0.5, 1, 2]:
print(a, (Lasso(alpha=a).fit(Xs, y).coef_ != 0).sum())
print(round(LassoCV(cv=5, random_state=0).fit(Xs, y).alpha_, 3))Expected output
0.01 18 0.1 8 0.5 3 1 2 2 1 0.076
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 What happens if alpha is chosen very large?
- The model overfits
- Nothing changes
- It errors
- The coefficients approach zero and the model underfits Correct answer
Explanation: When the penalty dominates, the model stops fitting the data; predictions approach the mean.
2 Set up the Lasso that chooses alpha with CV.
l = ____(cv=5, random_state=0).fit(Xs, y)l = LassoCV(cv=5, random_state=0).fit(Xs, y)Explanation: LassoCV generates its own alpha grid and stores the best in alpha_.
3 Match the tools.
| Term | Meaning |
|---|---|
RidgeCV |
Fast CV for Ridge |
LassoCV |
CV for Lasso |
alpha_ |
Chosen penalty |
lasso_path |
Coefficient path |