Summary - Feature selection
- Machine Learning course
- Intermediate
- Lesson 86/112
- 6 min
In this group you learned variance threshold, SelectKBest, and RFE.
Summary: feature selection
In this group:
- Selection keeps original columns; noise columns especially hurt distance-based models.
- Filters:
VarianceThreshold(doesn't look at the target),SelectKBest(f_classif, mutual_info). - Wrappers:
RFE,RFECV; evaluate features together, expensive. - Embedded: tree importances or Lasso coefficients with
SelectFromModel. - Every selection that looks at the target must be done inside a pipeline; k is a hyperparameter.
Practice Choose k with CV
Search selectkbest__k among [2, 4, 6, 10, 20] for make_pipeline(StandardScaler(), SelectKBest(f_classif), KNeighborsClassifier()) with GridSearchCV(..., cv=5). Print best_params_ and best_score_ (round(..., 3)) on separate lines.
Starter code
import numpy as np
from sklearn.datasets import make_classification
X, y = make_classification(n_samples=400, n_features=20, n_informative=4, n_redundant=2, shuffle=False, random_state=0)
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import GridSearchCV
# buraya yazSample solution
import numpy as np
from sklearn.datasets import make_classification
X, y = make_classification(n_samples=400, n_features=20, n_informative=4, n_redundant=2, shuffle=False, random_state=0)
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import GridSearchCV
pipe = make_pipeline(StandardScaler(), SelectKBest(f_classif), KNeighborsClassifier())
gs = GridSearchCV(pipe, {"selectkbest__k": [2, 4, 6, 10, 20]}, cv=5).fit(X, y)
print(gs.best_params_)
print(round(gs.best_score_, 3))Expected output
{'selectkbest__k': 4}
0.758
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 Which can be used safely outside a pipeline because it doesn't look at the target?
- SelectFromModel
- VarianceThreshold Correct answer
- SelectKBest
- RFE
Explanation: Variance is computed from X alone; the others use y.
2 Search the selection count in a grid.
Question
gs = GridSearchCV(pipe, {"____": [2, 4, 6, 10]}, cv=5)
Answer
gs = GridSearchCV(pipe, {"selectkbest__k": [2, 4, 6, 10]}, cv=5)Explanation: In make_pipeline the SelectKBest step is named "selectkbest".
3 Match the tools with the approaches.
| Term | Meaning |
|---|---|
VarianceThreshold |
Filter (no target) |
SelectKBest |
Filter (with target) |
RFE |
Wrapper |
SelectFromModel |
Embedded |