Challenge 1
- Machine Learning course
- Intermediate
- Lesson 109/112
- 8 min
Perform classification with scikit-learn.
Challenge 1: the full chain
The first challenge combining the intermediate tools: build a scaling + PCA + SVM chain on the digit data and search three hyperparameters together. The grid contains both preprocessing and model settings:
pipe = make_pipeline(StandardScaler(), PCA(), SVC())
izgara = {"pca__n_components": [20, 40],
"svc__C": [1, 10],
"svc__gamma": ["scale", 0.01]}
gs = GridSearchCV(pipe, izgara, cv=5, n_jobs=-1).fit(X_train, y_train)2 × 2 × 2 = 8 combinations, 40 trainings with 5 folds; the test set is used only at the very end.
Practice Search the chain
Run GridSearchCV(make_pipeline(StandardScaler(), PCA(), SVC()), izgara, cv=5) with the grid above. Print best_params_ and the test accuracy with round(..., 3) on separate lines.
Starter code
import numpy as np
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
X, y = load_digits(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.svm import SVC
from sklearn.model_selection import GridSearchCV
# buraya yazSample solution
import numpy as np
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
X, y = load_digits(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.svm import SVC
from sklearn.model_selection import GridSearchCV
izgara = {"pca__n_components": [20, 40], "svc__C": [1, 10], "svc__gamma": ["scale", 0.01]}
gs = GridSearchCV(make_pipeline(StandardScaler(), PCA(), SVC()), izgara, cv=5).fit(X_train, y_train)
print(gs.best_params_)
print(round(gs.score(X_test, y_test), 3))Expected output
{'pca__n_components': 40, 'svc__C': 10, 'svc__gamma': 0.01}
0.976
Practice Error analysis
Compute the confusion matrix with the best model's test predictions. Zero the diagonal (np.fill_diagonal(cm, 0)) and find the most confused (true, predicted) pair with np.unravel_index, printing it as a tuple of integers.
Starter code
import numpy as np
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
X, y = load_digits(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.svm import SVC
from sklearn.metrics import confusion_matrix
model = make_pipeline(StandardScaler(), PCA(n_components=40), SVC(C=10, gamma=0.01)).fit(X_train, y_train)
# buraya yazSample solution
import numpy as np
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
X, y = load_digits(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.svm import SVC
from sklearn.metrics import confusion_matrix
model = make_pipeline(StandardScaler(), PCA(n_components=40), SVC(C=10, gamma=0.01)).fit(X_train, y_train)
cm = confusion_matrix(y_test, model.predict(X_test))
np.fill_diagonal(cm, 0)
print(tuple(int(v) for v in np.unravel_index(cm.argmax(), cm.shape)))Expected output
(6, 4)
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 Why might this grid's best result still drop slightly on the test set?
- SVC is random
- It doesn't drop
- best_score_ is optimistic because of the selection Correct answer
- PCA corrupts the test data
Explanation: Picking the best of many combinations creates an upward bias.
2 Add the PCA component count to the grid.
izgara = {"____": [20, 40], "svc__C": [1, 10]}izgara = {"pca__n_components": [20, 40], "svc__C": [1, 10]}Explanation: In make_pipeline the PCA step is named "pca".
3 Match the chain parts.
| Term | Meaning |
|---|---|
StandardScaler |
Scaling |
PCA |
Dimensionality reduction |
SVC |
Classifier |
GridSearchCV |
Joint search |