Machine Learning
What is grid search?
Grid search tries all hyperparameter combinations to find the best.
Hyperparameter search
A model's success depends heavily on its hyperparameters: C and gamma in an SVM, max_depth in a tree. Trying by hand is both slow and prone to leakage. A systematic search measures every combination of settings with CV and picks the best.
for C in [0.1, 1, 10]:
s = cross_val_score(SVC(C=C), X, y, cv=5).mean()
print(C, round(s, 3))
# GridSearchCV does this loop (and more) in one line
The search is done only on the training set; the test set is for a single measurement at the very end.
Logarithmic grid
The effect of parameters like C, alpha, gamma and the learning rate changes by orders of magnitude. That's why the grid is built logarithmically, not linearly:
import numpy as np
print(np.logspace(-2, 2, 5)) # [ 0.01 0.1 1. 10. 100. ]
# bad: [1, 2, 3, 4, 5] -> all the same order of magnitude
First a wide logarithmic grid finds the right region, then a finer grid in that region fine-tunes.
Exercise Search by hand
Try C with the values np.logspace(-2, 2, 5): for each, compute the 5-fold CV mean of a scaled SVC. Print the best C and its score with round(..., 3) on one line.
Show example solution
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
X, y = load_breast_cancer(return_X_y=True)
skor = {}
for C in np.logspace(-2, 2, 5):
skor[C] = cross_val_score(make_pipeline(StandardScaler(), SVC(C=C)), X, y, cv=5).mean()
en_iyi = max(skor, key=skor.get)
print(en_iyi, round(skor[en_iyi], 3))
Test yourself
-
Which data should be used for the search?
- Only the training set (with CV inside)
- The test set
- All the data
- No validation is needed
Show answer
Only the training set (with CV inside) — If the test set takes part in the search, the final measurement isn't independent.
-
Produce 7 logarithmic values between 10⁻³ and 10³.
C_degerleri = np.____(-3, 3, 7)
Show answer
C_degerleri = np.logspace(-3, 3, 7)
-
Match the concepts.
- hyperparameter
- Setting chosen beforehand
- logspace
- Order-of-magnitude grid
- coarse grid
- Finds the region
- fine grid
- Fine-tuning
Next lesson: GridSearchCV
