Machine Learning
Feature selection
Select the best features.
Feature selection
We'll examine which of the 12 columns are needed with f_regression scores. The scores measure each column's linear relationship with the target on its own:
from sklearn.feature_selection import f_regression
F, p = f_regression(X_train, y_train)
print(np.argsort(-F)) # strongest to weakest
The noise columns usually get low scores and large p-values. But a real feature that looks weak on its own (floor with its small effect, or rooms whose effect comes through the ratio) can also drop to the bottom; univariate scores are a hint, not a final decision.
Searching k inside a pipeline
CV decides how many columns to keep. Ridge builds a lightly regularised linear model with the selected columns:
pipe = make_pipeline(StandardScaler(), SelectKBest(f_regression), Ridge(alpha=1.0))
gs = GridSearchCV(pipe, {"selectkbest__k": [3, 5, 7, 9, 12]}, cv=5,
scoring="neg_mean_absolute_error").fit(X_train, y_train)
print(gs.best_params_, -gs.best_score_)
The selected columns are read with gs.best_estimator_["selectkbest"].get_support(indices=True).
Exercise Find the noise
Run f_regression on the training data. Print the indices of the 5 weakest columns by F score, sorted ascending (np.sort(np.argsort(F)[:5])). Then print the smallest p-value of these columns with round(..., 2).
Show example solution
import numpy as np
rng = np.random.default_rng(11)
n = 500
m2 = rng.uniform(40, 200, n)
oda = np.clip(np.round(m2 / 35 + rng.normal(0, 0.7, n)), 1, 6)
yas = rng.uniform(0, 40, n)
kat = rng.integers(0, 15, n)
merkez_km = rng.uniform(0.5, 25, n)
gurultu = rng.normal(size=(n, 5))
fiyat = 20 * m2 + 150 * m2 / oda - 2500 * np.log1p(merkez_km) - 40 * yas + 60 * kat + rng.normal(0, 300, n)
X = np.column_stack([m2, oda, yas, kat, merkez_km, gurultu])
X2 = np.column_stack([X, m2 / oda, np.log1p(merkez_km)])
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X2, fiyat, test_size=0.25, random_state=0)
from sklearn.feature_selection import f_regression
F, p = f_regression(X_train, y_train)
zayif = np.sort(np.argsort(F)[:5])
print(zayif)
print(round(p[zayif].min(), 2))
Test yourself
-
What can be said of a column with a p-value of 0.6?
- It was computed wrongly
- The evidence for a linear relation with the target is weak
- It is a very strong feature
- It must be kept
Show answer
The evidence for a linear relation with the target is weak — A large p-value shows the observed relation can be explained by chance.
-
Compute the F scores for regression.
F, p = ____(X_train, y_train)
Show answer
F, p = f_regression(X_train, y_train)
-
Match the pieces.
- F score
- Strength of relation
- p-value
- Chance probability
- selectkbest__k
- Columns to keep
- get_support
- Selected columns
Next lesson: Training model
