Machine Learning
ElasticNet
ElasticNet combines L1 and L2 regularization.
ElasticNet
ElasticNet combines the L1 and L2 penalties. l1_ratio is the mixing ratio: 1 is pure Lasso, 0 pure Ridge. It offers Lasso's sparsity together with Ridge's stability on correlated features:
from sklearn.linear_model import ElasticNet, ElasticNetCV
en = ElasticNet(alpha=0.1, l1_ratio=0.5).fit(Xs, y)
encv = ElasticNetCV(l1_ratio=[0.1, 0.5, 0.9, 1.0], cv=5, random_state=0).fit(Xs, y)
print(encv.alpha_, encv.l1_ratio_)
With groups of correlated features, ElasticNet tends to keep the group together; Lasso picks a single one from the group.
Regularisation in classification
Logistic regression is regularised too; but in scikit-learn the penalty strength is given as C = 1/α: small C means a strong penalty. A suitable solver must be chosen for the L1 penalty:
LogisticRegression(C=0.1) # strong L2
LogisticRegression(penalty="l1", C=0.1, solver="liblinear") # sparse
LogisticRegression(penalty="elasticnet", l1_ratio=0.5, solver="saga", C=1)
The C in SVMs follows the same logic: large C means little penalty, a tight fit.
Exercise Grouped features
Train Lasso(alpha=0.1) and ElasticNet(alpha=0.1, l1_ratio=0.3) with G, made of three noisy copies of the first column. For each print the number of non-zero coefficients on one line (Lasso first).
Show example solution
import numpy as np
from sklearn.preprocessing import StandardScaler
rng = np.random.default_rng(0)
X = rng.normal(size=(60, 20))
y = 3 * X[:, 0] - 2 * X[:, 1] + 1.5 * X[:, 2] + rng.normal(0, 1, 60)
Xs = StandardScaler().fit_transform(X)
from sklearn.linear_model import Lasso, ElasticNet
G = np.column_stack([Xs[:, 0] + rng.normal(0, 0.05, 60) for _ in range(3)])
a = (Lasso(alpha=0.1).fit(G, y).coef_ != 0).sum()
b = (ElasticNet(alpha=0.1, l1_ratio=0.3).fit(G, y).coef_ != 0).sum()
print(a, b)
Test yourself
-
What is <code>ElasticNet(l1_ratio=1.0)</code> equal to?
- LinearRegression
- None of them
- Lasso
- Ridge
Show answer
Lasso — l1_ratio = 1 means the penalty is entirely L1.
-
Mix L1 and L2 equally.
en = ElasticNet(alpha=0.1, ____=0.5)
Show answer
en = ElasticNet(alpha=0.1, l1_ratio=0.5)
-
Match the parameters.
- l1_ratio=0
- Pure Ridge
- l1_ratio=1
- Pure Lasso
- small C
- Strong penalty
- solver="liblinear"
- L1 logistic
Next lesson: Summary - Regularization
