Machine Learning
Feature importance
Tree-based models compute feature importance.
Selection by importance: SelectFromModel
In the embedded approach the selection relies on the model's own measure. SelectFromModel keeps features whose importance is above a threshold (default: the mean importance):
from sklearn.feature_selection import SelectFromModel
sfm = SelectFromModel(RandomForestClassifier(n_estimators=200, random_state=0)).fit(X, y)
print(sfm.get_support(indices=True))
sfm2 = SelectFromModel(Lasso(alpha=0.05)).fit(Xs, y_reg) # non-zero coefficients
Lasso makes a natural selection by pushing coefficients to zero; you'll see this in detail in the regularisation lesson.
The problem of correlated features
If two features carry the same information, trees share the importance between them; both may fall below the threshold and be dropped. Lasso usually picks one and zeroes the other; which one it picks depends on the data:
# x and x + small noise: twin features
# forest: ~0.25 + ~0.25 (each looks small)
# lasso: 0.5 + 0.0 (one picked arbitrarily)
It is more accurate to interpret selection results not as "this feature is unnecessary" but as "this set gives enough success".
Exercise Selection with a forest
Train SelectFromModel(RandomForestClassifier(n_estimators=200, random_state=0)) on the synthetic data. Print the selected indices and the 5-fold CV accuracy of scaled logistic regression on the selected columns (round(..., 3)) on separate lines.
Show example solution
import numpy as np
from sklearn.datasets import make_classification
X, y = make_classification(n_samples=400, n_features=20, n_informative=4, n_redundant=2, shuffle=False, random_state=0)
from sklearn.ensemble import RandomForestClassifier
from sklearn.feature_selection import SelectFromModel
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
sfm = SelectFromModel(RandomForestClassifier(n_estimators=200, random_state=0)).fit(X, y)
idx = sfm.get_support(indices=True)
print(idx)
m = make_pipeline(StandardScaler(), LogisticRegression())
print(round(cross_val_score(m, X[:, idx], y, cv=5).mean(), 3))
Test yourself
-
What is SelectFromModel's default threshold?
- The median
- The largest importance
- The mean of the importances
- Zero
Show answer
The mean of the importances — It can be changed with values like threshold="median" or "1.5*mean".
-
Keep those with importance above the median.
sfm = SelectFromModel(model, threshold=____)
Show answer
sfm = SelectFromModel(model, threshold="median")
-
Match the methods.
- SelectFromModel
- Selection by importance threshold
- Lasso
- Zeroes coefficients
- twin features
- Share importance
- threshold
- Selection limit
Next lesson: Summary - Feature selection
