Challenge 4
- Machine Learning course
- Beginner
- Lesson 112/112
- 8 min
Evaluate a classification model.
Challenge 4: end-to-end project
In the final challenge you'll combine every step of the beginner level: split the data, build a pipeline, choose a hyperparameter with cross-validation, measure once on the test set and report per class.
izgara = {"kneighborsclassifier__n_neighbors": [1, 3, 5, 7, 9]}
gs = GridSearchCV(make_pipeline(StandardScaler(), KNeighborsClassifier()), izgara, cv=5)
gs.fit(X_train, y_train)
print(gs.best_params_, gs.score(X_test, y_test))Because make_pipeline names steps with the lowercase class name, the parameter becomes kneighborsclassifier__n_neighbors.
Practice Search and measure
Split the wine data with test_size=0.25, random_state=0, stratify=y. Search n_neighbors among [1, 3, 5, 7, 9] for make_pipeline(StandardScaler(), KNeighborsClassifier()) with GridSearchCV(..., cv=5). Print best_params_ and the test accuracy (round(..., 3)) on separate lines.
Starter code
from sklearn.datasets import load_wine
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
X, y = load_wine(return_X_y=True)
# buraya yazSample solution
from sklearn.datasets import load_wine
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
X, y = load_wine(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
izgara = {"kneighborsclassifier__n_neighbors": [1, 3, 5, 7, 9]}
gs = GridSearchCV(make_pipeline(StandardScaler(), KNeighborsClassifier()), izgara, cv=5)
gs.fit(X_train, y_train)
print(gs.best_params_)
print(round(gs.score(X_test, y_test), 3))Expected output
{'kneighborsclassifier__n_neighbors': 9}
0.956
Practice Per-class report
Get the test predictions with the best model. Print the per-class F1 scores with f1_score(..., average=None).round(3), then the macro F1 with round(..., 3).
Starter code
from sklearn.datasets import load_wine
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import f1_score
X, y = load_wine(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
model = make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=9)).fit(X_train, y_train)
# buraya yazSample solution
from sklearn.datasets import load_wine
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import f1_score
X, y = load_wine(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
model = make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=9)).fit(X_train, y_train)
p = model.predict(X_test)
print(f1_score(y_test, p, average=None).round(3))
print(round(f1_score(y_test, p, average="macro"), 3))Expected output
[0.968 0.941 0.96 ] 0.956
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 How many times is the test set used in this flow?
- For every parameter
- Never
- Once, at the very end Correct answer
- In every CV fold
Explanation: Selection is done with CV on the training set; the test is only the final measurement.
2 Put the neighbour count of the KNN inside the pipeline into the grid.
izgara = {"____": [3, 5, 7]}izgara = {"kneighborsclassifier__n_neighbors": [3, 5, 7]}Explanation: step__parameter; the make_pipeline name is the lowercase class name.
3 Match the steps with the tools.
| Term | Meaning |
|---|---|
Splitting |
train_test_split |
Preprocessing + model |
make_pipeline |
Setting selection |
GridSearchCV |
Per-class report |
classification_report |