Machine Learning
Challenge 1
Perform classification with scikit-learn.
Challenge 1: the full chain
The first challenge combining the intermediate tools: build a scaling + PCA + SVM chain on the digit data and search three hyperparameters together. The grid contains both preprocessing and model settings:
pipe = make_pipeline(StandardScaler(), PCA(), SVC())
izgara = {"pca__n_components": [20, 40],
"svc__C": [1, 10],
"svc__gamma": ["scale", 0.01]}
gs = GridSearchCV(pipe, izgara, cv=5, n_jobs=-1).fit(X_train, y_train)
2 × 2 × 2 = 8 combinations, 40 trainings with 5 folds; the test set is used only at the very end.
Exercise Search the chain
Run GridSearchCV(make_pipeline(StandardScaler(), PCA(), SVC()), izgara, cv=5) with the grid above. Print best_params_ and the test accuracy with round(..., 3) on separate lines.
Show example solution
import numpy as np
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
X, y = load_digits(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.svm import SVC
from sklearn.model_selection import GridSearchCV
izgara = {"pca__n_components": [20, 40], "svc__C": [1, 10], "svc__gamma": ["scale", 0.01]}
gs = GridSearchCV(make_pipeline(StandardScaler(), PCA(), SVC()), izgara, cv=5).fit(X_train, y_train)
print(gs.best_params_)
print(round(gs.score(X_test, y_test), 3))
Exercise Error analysis
Compute the confusion matrix with the best model's test predictions. Zero the diagonal (np.fill_diagonal(cm, 0)) and find the most confused (true, predicted) pair with np.unravel_index, printing it as a tuple of integers.
Show example solution
import numpy as np
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
X, y = load_digits(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.svm import SVC
from sklearn.metrics import confusion_matrix
model = make_pipeline(StandardScaler(), PCA(n_components=40), SVC(C=10, gamma=0.01)).fit(X_train, y_train)
cm = confusion_matrix(y_test, model.predict(X_test))
np.fill_diagonal(cm, 0)
print(tuple(int(v) for v in np.unravel_index(cm.argmax(), cm.shape)))
Test yourself
-
Why might this grid's best result still drop slightly on the test set?
- SVC is random
- It doesn't drop
- best_score_ is optimistic because of the selection
- PCA corrupts the test data
Show answer
best_score_ is optimistic because of the selection — Picking the best of many combinations creates an upward bias.
-
Add the PCA component count to the grid.
izgara = {"____": [20, 40], "svc__C": [1, 10]}
Show answer
izgara = {"pca__n_components": [20, 40], "svc__C": [1, 10]}
-
Match the chain parts.
- StandardScaler
- Scaling
- PCA
- Dimensionality reduction
- SVC
- Classifier
- GridSearchCV
- Joint search
Next lesson: Challenge 2
