Machine Learning
What is feature selection?
Feature selection simplifies the model by choosing the most informative features.
What is feature selection?
Feature selection drops columns that don't contribute to the model's success and keeps the useful ones. Its difference from PCA is that the result consists of original, interpretable columns. There are three approaches:
- Filter: model-independent statistics (variance, correlation, F test).
- Wrapper: searches for a subset by training a model repeatedly (RFE).
- Embedded: the model's own importances (Lasso coefficients, tree importances).
Synthetic test data
Data where we know which columns are useful is ideal for testing selection methods. make_classification with shuffle=False puts the useful columns first:
X, y = make_classification(n_samples=400, n_features=20, n_informative=4,
n_redundant=2, shuffle=False, random_state=0)
# columns 0-3: informative, 4-5: linear combinations of them, 6-19: noise
A good selection method should pick the first 6 columns and drop the remaining 14 noise columns.
Exercise The harm of noise
For KNN (5 neighbours, scaled), compute the 5-fold CV accuracy with only the first 6 columns and with all 20 columns. Print the two scores with round(..., 3) on one line (6 columns first).
Show example solution
import numpy as np
from sklearn.datasets import make_classification
X, y = make_classification(n_samples=400, n_features=20, n_informative=4, n_redundant=2, shuffle=False, random_state=0)
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import cross_val_score
m = make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=5))
print(round(cross_val_score(m, X[:, :6], y, cv=5).mean(), 3), round(cross_val_score(m, X, y, cv=5).mean(), 3))
Test yourself
-
Which is a wrapper method?
- Correlation filter
- PCA
- Recursive feature elimination
- Variance threshold
Show answer
Recursive feature elimination — RFE trains the model repeatedly for the selection.
-
Set the number of informative features to 4.
X, y = make_classification(n_features=20, ____=4, random_state=0)
Show answer
X, y = make_classification(n_features=20, n_informative=4, random_state=0)
-
Match the approaches.
- filter
- Model-independent statistic
- wrapper
- Repeated model training
- embedded
- The model's own importance
- n_redundant
- Repeated information
Next lesson: Variance threshold
