What is feature selection?
- Machine Learning course
- Intermediate
- Lesson 81/112
- 5 min
Feature selection simplifies the model by choosing the most informative features.
What is feature selection?
Feature selection drops columns that don't contribute to the model's success and keeps the useful ones. Its difference from PCA is that the result consists of original, interpretable columns. There are three approaches:
- Filter: model-independent statistics (variance, correlation, F test).
- Wrapper: searches for a subset by training a model repeatedly (RFE).
- Embedded: the model's own importances (Lasso coefficients, tree importances).
Synthetic test data
Data where we know which columns are useful is ideal for testing selection methods. make_classification with shuffle=False puts the useful columns first:
X, y = make_classification(n_samples=400, n_features=20, n_informative=4,
n_redundant=2, shuffle=False, random_state=0)
# columns 0-3: informative, 4-5: linear combinations of them, 6-19: noiseA good selection method should pick the first 6 columns and drop the remaining 14 noise columns.
Practice The harm of noise
For KNN (5 neighbours, scaled), compute the 5-fold CV accuracy with only the first 6 columns and with all 20 columns. Print the two scores with round(..., 3) on one line (6 columns first).
Starter code
import numpy as np
from sklearn.datasets import make_classification
X, y = make_classification(n_samples=400, n_features=20, n_informative=4, n_redundant=2, shuffle=False, random_state=0)
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import cross_val_score
# buraya yazSample solution
import numpy as np
from sklearn.datasets import make_classification
X, y = make_classification(n_samples=400, n_features=20, n_informative=4, n_redundant=2, shuffle=False, random_state=0)
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import cross_val_score
m = make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=5))
print(round(cross_val_score(m, X[:, :6], y, cv=5).mean(), 3), round(cross_val_score(m, X, y, cv=5).mean(), 3))Expected output
0.878 0.688
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 Which is a wrapper method?
- Correlation filter
- PCA
- Recursive feature elimination Correct answer
- Variance threshold
Explanation: RFE trains the model repeatedly for the selection.
2 Set the number of informative features to 4.
X, y = make_classification(n_features=20, ____=4, random_state=0)X, y = make_classification(n_features=20, n_informative=4, random_state=0)Explanation: n_informative is the number of columns that really determine the class.
3 Match the approaches.
| Term | Meaning |
|---|---|
filter |
Model-independent statistic |
wrapper |
Repeated model training |
embedded |
The model's own importance |
n_redundant |
Repeated information |