Machine Learning
Challenge 4
Evaluate a classification model.
Challenge 4: end-to-end project
In the final challenge you'll combine every step of the beginner level: split the data, build a pipeline, choose a hyperparameter with cross-validation, measure once on the test set and report per class.
izgara = {"kneighborsclassifier__n_neighbors": [1, 3, 5, 7, 9]}
gs = GridSearchCV(make_pipeline(StandardScaler(), KNeighborsClassifier()), izgara, cv=5)
gs.fit(X_train, y_train)
print(gs.best_params_, gs.score(X_test, y_test))
Because make_pipeline names steps with the lowercase class name, the parameter becomes kneighborsclassifier__n_neighbors.
Exercise Search and measure
Split the wine data with test_size=0.25, random_state=0, stratify=y. Search n_neighbors among [1, 3, 5, 7, 9] for make_pipeline(StandardScaler(), KNeighborsClassifier()) with GridSearchCV(..., cv=5). Print best_params_ and the test accuracy (round(..., 3)) on separate lines.
Show example solution
from sklearn.datasets import load_wine
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
X, y = load_wine(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
izgara = {"kneighborsclassifier__n_neighbors": [1, 3, 5, 7, 9]}
gs = GridSearchCV(make_pipeline(StandardScaler(), KNeighborsClassifier()), izgara, cv=5)
gs.fit(X_train, y_train)
print(gs.best_params_)
print(round(gs.score(X_test, y_test), 3))
Exercise Per-class report
Get the test predictions with the best model. Print the per-class F1 scores with f1_score(..., average=None).round(3), then the macro F1 with round(..., 3).
Show example solution
from sklearn.datasets import load_wine
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import f1_score
X, y = load_wine(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
model = make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=9)).fit(X_train, y_train)
p = model.predict(X_test)
print(f1_score(y_test, p, average=None).round(3))
print(round(f1_score(y_test, p, average="macro"), 3))
Test yourself
-
How many times is the test set used in this flow?
- For every parameter
- Never
- Once, at the very end
- In every CV fold
Show answer
Once, at the very end — Selection is done with CV on the training set; the test is only the final measurement.
-
Put the neighbour count of the KNN inside the pipeline into the grid.
izgara = {"____": [3, 5, 7]}
Show answer
izgara = {"kneighborsclassifier__n_neighbors": [3, 5, 7]}
-
Match the steps with the tools.
- Splitting
- train_test_split
- Preprocessing + model
- make_pipeline
- Setting selection
- GridSearchCV
- Per-class report
- classification_report
Next lesson: Beginner section exam
