Feature selection
- Machine Learning course
- Intermediate
- Lesson 94/112
- 7 min
Select the best features.
Feature selection
We'll examine which of the 12 columns are needed with f_regression scores. The scores measure each column's linear relationship with the target on its own:
from sklearn.feature_selection import f_regression
F, p = f_regression(X_train, y_train)
print(np.argsort(-F)) # strongest to weakestThe noise columns usually get low scores and large p-values. But a real feature that looks weak on its own (floor with its small effect, or rooms whose effect comes through the ratio) can also drop to the bottom; univariate scores are a hint, not a final decision.
Searching k inside a pipeline
CV decides how many columns to keep. Ridge builds a lightly regularised linear model with the selected columns:
pipe = make_pipeline(StandardScaler(), SelectKBest(f_regression), Ridge(alpha=1.0))
gs = GridSearchCV(pipe, {"selectkbest__k": [3, 5, 7, 9, 12]}, cv=5,
scoring="neg_mean_absolute_error").fit(X_train, y_train)
print(gs.best_params_, -gs.best_score_)The selected columns are read with gs.best_estimator_["selectkbest"].get_support(indices=True).
Practice Find the noise
Run f_regression on the training data. Print the indices of the 5 weakest columns by F score, sorted ascending (np.sort(np.argsort(F)[:5])). Then print the smallest p-value of these columns with round(..., 2).
Starter code
import numpy as np
rng = np.random.default_rng(11)
n = 500
m2 = rng.uniform(40, 200, n)
oda = np.clip(np.round(m2 / 35 + rng.normal(0, 0.7, n)), 1, 6)
yas = rng.uniform(0, 40, n)
kat = rng.integers(0, 15, n)
merkez_km = rng.uniform(0.5, 25, n)
gurultu = rng.normal(size=(n, 5))
fiyat = 20 * m2 + 150 * m2 / oda - 2500 * np.log1p(merkez_km) - 40 * yas + 60 * kat + rng.normal(0, 300, n)
X = np.column_stack([m2, oda, yas, kat, merkez_km, gurultu])
X2 = np.column_stack([X, m2 / oda, np.log1p(merkez_km)])
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X2, fiyat, test_size=0.25, random_state=0)
from sklearn.feature_selection import f_regression
# buraya yazSample solution
import numpy as np
rng = np.random.default_rng(11)
n = 500
m2 = rng.uniform(40, 200, n)
oda = np.clip(np.round(m2 / 35 + rng.normal(0, 0.7, n)), 1, 6)
yas = rng.uniform(0, 40, n)
kat = rng.integers(0, 15, n)
merkez_km = rng.uniform(0.5, 25, n)
gurultu = rng.normal(size=(n, 5))
fiyat = 20 * m2 + 150 * m2 / oda - 2500 * np.log1p(merkez_km) - 40 * yas + 60 * kat + rng.normal(0, 300, n)
X = np.column_stack([m2, oda, yas, kat, merkez_km, gurultu])
X2 = np.column_stack([X, m2 / oda, np.log1p(merkez_km)])
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X2, fiyat, test_size=0.25, random_state=0)
from sklearn.feature_selection import f_regression
F, p = f_regression(X_train, y_train)
zayif = np.sort(np.argsort(F)[:5])
print(zayif)
print(round(p[zayif].min(), 2))Expected output
[3 5 6 7 9] 0.19
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 What can be said of a column with a p-value of 0.6?
- It was computed wrongly
- The evidence for a linear relation with the target is weak Correct answer
- It is a very strong feature
- It must be kept
Explanation: A large p-value shows the observed relation can be explained by chance.
2 Compute the F scores for regression.
F, p = ____(X_train, y_train)F, p = f_regression(X_train, y_train)Explanation: f_regression produces an F statistic from each column's correlation with the target.
3 Match the pieces.
| Term | Meaning |
|---|---|
F score |
Strength of relation |
p-value |
Chance probability |
selectkbest__k |
Columns to keep |
get_support |
Selected columns |