Machine Learning
Training model
Train a model with selected features.
Model training
We combine selection and the model in one pipeline and search k by MAE. For comparison we also train the same chain without the derived features:
gs = GridSearchCV(pipe, {"selectkbest__k": [3, 5, 7, 9, 12]}, cv=5,
scoring="neg_mean_absolute_error").fit(X_train, y_train)
The test set stays untouched throughout the project; all decisions are made with CV on the training data.
Reading the selected columns
Seeing which columns the best model uses checks whether the selection makes sense:
sec = gs.best_estimator_["selectkbest"]
print(sec.get_support(indices=True))
On this data CV selects all 12 columns: since Ridge's penalty already keeps the noise columns' coefficients small, dropping them doesn't improve the score, while dropping real but weak-looking columns makes it worse. The value of selection shows up with many noise columns or with noise-sensitive models like KNN. In this project the main gain will come from the derived features.
Exercise Search and inspect
Search selectkbest__k among [3, 5, 7, 9, 12] for make_pipeline(StandardScaler(), SelectKBest(f_regression), Ridge(alpha=1.0)) with GridSearchCV(..., cv=5, scoring="neg_mean_absolute_error"). Print best_params_, the CV MAE (round(-gs.best_score_, 1)) and the selected column indices on separate lines.
Show example solution
import numpy as np
rng = np.random.default_rng(11)
n = 500
m2 = rng.uniform(40, 200, n)
oda = np.clip(np.round(m2 / 35 + rng.normal(0, 0.7, n)), 1, 6)
yas = rng.uniform(0, 40, n)
kat = rng.integers(0, 15, n)
merkez_km = rng.uniform(0.5, 25, n)
gurultu = rng.normal(size=(n, 5))
fiyat = 20 * m2 + 150 * m2 / oda - 2500 * np.log1p(merkez_km) - 40 * yas + 60 * kat + rng.normal(0, 300, n)
X = np.column_stack([m2, oda, yas, kat, merkez_km, gurultu])
X2 = np.column_stack([X, m2 / oda, np.log1p(merkez_km)])
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X2, fiyat, test_size=0.25, random_state=0)
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.feature_selection import SelectKBest, f_regression
from sklearn.linear_model import Ridge
from sklearn.model_selection import GridSearchCV
pipe = make_pipeline(StandardScaler(), SelectKBest(f_regression), Ridge(alpha=1.0))
gs = GridSearchCV(pipe, {"selectkbest__k": [3, 5, 7, 9, 12]}, cv=5, scoring="neg_mean_absolute_error").fit(X_train, y_train)
print(gs.best_params_)
print(round(-gs.best_score_, 1))
print(gs.best_estimator_["selectkbest"].get_support(indices=True))
Test yourself
-
Why is <code>scoring="neg_mean_absolute_error"</code> written rather than "mean_absolute_error"?
- scikit-learn treats a larger score as better; the error is negated
- MAE can't be computed
- It is faster
- There is no difference
Show answer
scikit-learn treats a larger score as better; the error is negated — best_score_ is the negative MAE; the sign is flipped when reporting.
-
Get the selector step in the best model.
sec = gs.best_estimator_[____]
Show answer
sec = gs.best_estimator_["selectkbest"]
-
Match the decisions.
- k
- Chosen with CV
- MAE
- Selection criterion
- test set
- Untouched until the end
- selected columns
- Sanity check
Next lesson: Evaluation
