What is grid search?
- Machine Learning course
- Intermediate
- Lesson 19/112
- 5 min
Grid search tries all hyperparameter combinations to find the best.
Hyperparameter search
A model's success depends heavily on its hyperparameters: C and gamma in an SVM, max_depth in a tree. Trying by hand is both slow and prone to leakage. A systematic search measures every combination of settings with CV and picks the best.
for C in [0.1, 1, 10]:
s = cross_val_score(SVC(C=C), X, y, cv=5).mean()
print(C, round(s, 3))
# GridSearchCV does this loop (and more) in one lineThe search is done only on the training set; the test set is for a single measurement at the very end.
Logarithmic grid
The effect of parameters like C, alpha, gamma and the learning rate changes by orders of magnitude. That's why the grid is built logarithmically, not linearly:
import numpy as np
print(np.logspace(-2, 2, 5)) # [ 0.01 0.1 1. 10. 100. ]
# bad: [1, 2, 3, 4, 5] -> all the same order of magnitudeFirst a wide logarithmic grid finds the right region, then a finer grid in that region fine-tunes.
Practice Search by hand
Try C with the values np.logspace(-2, 2, 5): for each, compute the 5-fold CV mean of a scaled SVC. Print the best C and its score with round(..., 3) on one line.
Starter code
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
X, y = load_breast_cancer(return_X_y=True)
# buraya yazSample solution
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
X, y = load_breast_cancer(return_X_y=True)
skor = {}
for C in np.logspace(-2, 2, 5):
skor[C] = cross_val_score(make_pipeline(StandardScaler(), SVC(C=C)), X, y, cv=5).mean()
en_iyi = max(skor, key=skor.get)
print(en_iyi, round(skor[en_iyi], 3))Expected output
10.0 0.977
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 Which data should be used for the search?
- Only the training set (with CV inside) Correct answer
- The test set
- All the data
- No validation is needed
Explanation: If the test set takes part in the search, the final measurement isn't independent.
2 Produce 7 logarithmic values between 10⁻³ and 10³.
C_degerleri = np.____(-3, 3, 7)C_degerleri = np.logspace(-3, 3, 7)Explanation: logspace takes exponents: 10^-3 ... 10^3.
3 Match the concepts.
| Term | Meaning |
|---|---|
hyperparameter |
Setting chosen beforehand |
logspace |
Order-of-magnitude grid |
coarse grid |
Finds the region |
fine grid |
Fine-tuning |