Recursive elimination
- Machine Learning course
- Intermediate
- Lesson 84/112
- 5 min
RFE recursively removes the least important feature.
Recursive feature elimination
RFE trains a model with all features, drops the least important feature (smallest coefficient or importance) and repeats down to the desired number. Because it evaluates features together, it is smarter than univariate filters:
from sklearn.feature_selection import RFE
rfe = RFE(LogisticRegression(max_iter=1000), n_features_to_select=6, step=1).fit(Xs, y)
print(rfe.support_)
print(rfe.ranking_) # 1 = selected, larger number = dropped earlierBecause it relies on model coefficients, scaling matters; step is how many features are dropped per round.
RFECV
If the number of features to keep is unknown, RFECV measures the CV score at every step and picks the best number itself:
from sklearn.feature_selection import RFECV
rc = RFECV(LogisticRegression(max_iter=1000), step=1, cv=5).fit(Xs, y)
print(rc.n_features_)
print(rc.cv_results_["mean_test_score"].round(3))It is expensive: CV is run for every feature count. With many columns step is increased.
Practice Selection with RFE
Train RFE(LogisticRegression(max_iter=1000), n_features_to_select=6) on the scaled synthetic data. Print the selected indices (np.where(rfe.support_)[0]) and how many of them are among the first 6 columns on separate lines.
Starter code
import numpy as np
from sklearn.datasets import make_classification
X, y = make_classification(n_samples=400, n_features=20, n_informative=4, n_redundant=2, shuffle=False, random_state=0)
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.feature_selection import RFE
Xs = StandardScaler().fit_transform(X)
# buraya yazSample solution
import numpy as np
from sklearn.datasets import make_classification
X, y = make_classification(n_samples=400, n_features=20, n_informative=4, n_redundant=2, shuffle=False, random_state=0)
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.feature_selection import RFE
Xs = StandardScaler().fit_transform(X)
rfe = RFE(LogisticRegression(max_iter=1000), n_features_to_select=6).fit(Xs, y)
idx = np.where(rfe.support_)[0]
print(idx)
print((idx < 6).sum())Expected output
[ 0 1 2 3 4 19] 5
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 What does a feature with <code>ranking_</code> value 1 mean?
- It was dropped first
- It is the worst
- It wasn't computed
- It is among the selected features Correct answer
Explanation: All selected ones get rank 1; dropped ones grow by drop order.
2 Drop 2 features per round.
rfe = RFE(model, n_features_to_select=5, ____=2)rfe = RFE(model, n_features_to_select=5, step=2)Explanation: step can be an integer or a ratio.
3 Match the attributes.
| Term | Meaning |
|---|---|
support_ |
Mask of selected |
ranking_ |
Drop order |
n_features_ |
Count chosen by RFECV |
step |
Dropped per round |