Summary - Feature selection

In this group you learned variance threshold, SelectKBest, and RFE.

Summary: feature selection

In this group:

Selection keeps original columns; noise columns especially hurt distance-based models.
Filters: VarianceThreshold (doesn't look at the target), SelectKBest (f_classif, mutual_info).
Wrappers: RFE, RFECV; evaluate features together, expensive.
Embedded: tree importances or Lasso coefficients with SelectFromModel.
Every selection that looks at the target must be done inside a pipeline; k is a hyperparameter.

Exercise Choose k with CV

Search selectkbest__k among [2, 4, 6, 10, 20] for make_pipeline(StandardScaler(), SelectKBest(f_classif), KNeighborsClassifier()) with GridSearchCV(..., cv=5). Print best_params_ and best_score_ (round(..., 3)) on separate lines.

Show example solution

import numpy as np
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=400, n_features=20, n_informative=4, n_redundant=2, shuffle=False, random_state=0)
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import GridSearchCV

pipe = make_pipeline(StandardScaler(), SelectKBest(f_classif), KNeighborsClassifier())
gs = GridSearchCV(pipe, {"selectkbest__k": [2, 4, 6, 10, 20]}, cv=5).fit(X, y)
print(gs.best_params_)
print(round(gs.best_score_, 3))

Test yourself

Which can be used safely outside a pipeline because it doesn't look at the target?
- SelectFromModel
- VarianceThreshold
- SelectKBest
- RFE
Show answer

VarianceThreshold — Variance is computed from X alone; the others use y.

Search the selection count in a grid.

gs = GridSearchCV(pipe, {"____": [2, 4, 6, 10]}, cv=5)

Show answer

gs = GridSearchCV(pipe, {"selectkbest__k": [2, 4, 6, 10]}, cv=5)

Match the tools with the approaches.

VarianceThreshold

Filter (no target)
SelectKBest

Filter (with target)
RFE

Wrapper
SelectFromModel

Embedded

Next lesson: What is imbalanced data?

We Are Enhancing Your Experience

Cookie Preferences

Strictly Necessary Cookies

Analytical Cookies