Feature importance
- Machine Learning course
- Intermediate
- Lesson 85/112
- 5 min
Tree-based models compute feature importance.
Selection by importance: SelectFromModel
In the embedded approach the selection relies on the model's own measure. SelectFromModel keeps features whose importance is above a threshold (default: the mean importance):
from sklearn.feature_selection import SelectFromModel
sfm = SelectFromModel(RandomForestClassifier(n_estimators=200, random_state=0)).fit(X, y)
print(sfm.get_support(indices=True))
sfm2 = SelectFromModel(Lasso(alpha=0.05)).fit(Xs, y_reg) # non-zero coefficientsLasso makes a natural selection by pushing coefficients to zero; you'll see this in detail in the regularisation lesson.
The problem of correlated features
If two features carry the same information, trees share the importance between them; both may fall below the threshold and be dropped. Lasso usually picks one and zeroes the other; which one it picks depends on the data:
# x and x + small noise: twin features
# forest: ~0.25 + ~0.25 (each looks small)
# lasso: 0.5 + 0.0 (one picked arbitrarily)It is more accurate to interpret selection results not as "this feature is unnecessary" but as "this set gives enough success".
Practice Selection with a forest
Train SelectFromModel(RandomForestClassifier(n_estimators=200, random_state=0)) on the synthetic data. Print the selected indices and the 5-fold CV accuracy of scaled logistic regression on the selected columns (round(..., 3)) on separate lines.
Starter code
import numpy as np
from sklearn.datasets import make_classification
X, y = make_classification(n_samples=400, n_features=20, n_informative=4, n_redundant=2, shuffle=False, random_state=0)
from sklearn.ensemble import RandomForestClassifier
from sklearn.feature_selection import SelectFromModel
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
# buraya yazSample solution
import numpy as np
from sklearn.datasets import make_classification
X, y = make_classification(n_samples=400, n_features=20, n_informative=4, n_redundant=2, shuffle=False, random_state=0)
from sklearn.ensemble import RandomForestClassifier
from sklearn.feature_selection import SelectFromModel
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
sfm = SelectFromModel(RandomForestClassifier(n_estimators=200, random_state=0)).fit(X, y)
idx = sfm.get_support(indices=True)
print(idx)
m = make_pipeline(StandardScaler(), LogisticRegression())
print(round(cross_val_score(m, X[:, idx], y, cv=5).mean(), 3))Expected output
[0 1 2 3 4 5] 0.653
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 What is SelectFromModel's default threshold?
- The median
- The largest importance
- The mean of the importances Correct answer
- Zero
Explanation: It can be changed with values like threshold="median" or "1.5*mean".
2 Keep those with importance above the median.
sfm = SelectFromModel(model, threshold=____)sfm = SelectFromModel(model, threshold="median")Explanation: A median threshold keeps half of the features.
3 Match the methods.
| Term | Meaning |
|---|---|
SelectFromModel |
Selection by importance threshold |
Lasso |
Zeroes coefficients |
twin features |
Share importance |
threshold |
Selection limit |