Machine Learning
SelectKBest
SelectKBest selects the top k features based on a statistical test.
SelectKBest
SelectKBest scores each feature one by one on its relationship with the target and keeps the best k. The scoring function is chosen per problem:
from sklearn.feature_selection import SelectKBest, f_classif, mutual_info_classif
sec = SelectKBest(score_func=f_classif, k=6).fit(X, y)
print(sec.get_support(indices=True))
print(sec.scores_.round(1))
f_classif (ANOVA F) measures linear separation, mutual_info_classif any kind of dependence. For regression f_regression and mutual_info_regression are used.
Pipeline and choosing k
Because SelectKBest looks at the target, using it outside a pipeline creates serious leakage (remember the random-data example in the pipeline lesson). k is a hyperparameter too:
pipe = make_pipeline(StandardScaler(), SelectKBest(f_classif), LogisticRegression())
gs = GridSearchCV(pipe, {"selectkbest__k": [2, 4, 6, 10, 20]}, cv=5).fit(X, y)
print(gs.best_params_)
That way the selection in each fold is made only with that fold's training data.
Exercise The best 6 features
Train SelectKBest(f_classif, k=6) on the synthetic data; print the selected column indices. Then print how many of the selected ones are really among the first 6 columns.
Show example solution
import numpy as np
from sklearn.datasets import make_classification
X, y = make_classification(n_samples=400, n_features=20, n_informative=4, n_redundant=2, shuffle=False, random_state=0)
from sklearn.feature_selection import SelectKBest, f_classif
sec = SelectKBest(f_classif, k=6).fit(X, y)
idx = sec.get_support(indices=True)
print(idx)
print((idx < 6).sum())
Test yourself
-
What is the weakness of univariate filters?
- They can miss features that carry information only together
- They are very slow
- They don't look at the target
- They don't work for classification
Show answer
They can miss features that carry information only together — Each feature is scored separately; interactions like XOR are invisible.
-
Score with mutual information.
sec = SelectKBest(score_func=____, k=5)
Show answer
sec = SelectKBest(score_func=mutual_info_classif, k=5)
-
Match the scoring functions.
- f_classif
- ANOVA F, classification
- f_regression
- Correlation-based, regression
- mutual_info_classif
- Any dependence
- chi2
- Non-negative counts
Next lesson: Recursive elimination
