Machine Learning
Recursive elimination
RFE recursively removes the least important feature.
Recursive feature elimination
RFE trains a model with all features, drops the least important feature (smallest coefficient or importance) and repeats down to the desired number. Because it evaluates features together, it is smarter than univariate filters:
from sklearn.feature_selection import RFE
rfe = RFE(LogisticRegression(max_iter=1000), n_features_to_select=6, step=1).fit(Xs, y)
print(rfe.support_)
print(rfe.ranking_) # 1 = selected, larger number = dropped earlier
Because it relies on model coefficients, scaling matters; step is how many features are dropped per round.
RFECV
If the number of features to keep is unknown, RFECV measures the CV score at every step and picks the best number itself:
from sklearn.feature_selection import RFECV
rc = RFECV(LogisticRegression(max_iter=1000), step=1, cv=5).fit(Xs, y)
print(rc.n_features_)
print(rc.cv_results_["mean_test_score"].round(3))
It is expensive: CV is run for every feature count. With many columns step is increased.
Exercise Selection with RFE
Train RFE(LogisticRegression(max_iter=1000), n_features_to_select=6) on the scaled synthetic data. Print the selected indices (np.where(rfe.support_)[0]) and how many of them are among the first 6 columns on separate lines.
Show example solution
import numpy as np
from sklearn.datasets import make_classification
X, y = make_classification(n_samples=400, n_features=20, n_informative=4, n_redundant=2, shuffle=False, random_state=0)
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.feature_selection import RFE
Xs = StandardScaler().fit_transform(X)
rfe = RFE(LogisticRegression(max_iter=1000), n_features_to_select=6).fit(Xs, y)
idx = np.where(rfe.support_)[0]
print(idx)
print((idx < 6).sum())
Test yourself
-
What does a feature with <code>ranking_</code> value 1 mean?
- It was dropped first
- It is the worst
- It wasn't computed
- It is among the selected features
Show answer
It is among the selected features — All selected ones get rank 1; dropped ones grow by drop order.
-
Drop 2 features per round.
rfe = RFE(model, n_features_to_select=5, ____=2)
Show answer
rfe = RFE(model, n_features_to_select=5, step=2)
-
Match the attributes.
- support_
- Mask of selected
- ranking_
- Drop order
- n_features_
- Count chosen by RFECV
- step
- Dropped per round
Next lesson: Feature importance
