best_score_
- Machine Learning course
- Intermediate
- Lesson 23/112
- 5 min
The best_score_ property gives the best cross-validation score.
Why is best_score_ optimistic?
best_score_ is the CV mean of the best combination. Since the highest of many combinations is picked, it tends to be a lucky result; the more combinations tried, the more optimism.
print(gs.best_score_) # the score of the selection: optimistic
print(gs.score(X_test, y_test)) # independent score: honestThe success to report is the test score or a nested CV score; best_score_ is only for ranking candidates.
RandomizedSearchCV
As the grid grows, the time explodes. RandomizedSearchCV tries n_iter random combinations from distributions; with the same budget it often does as well as a grid, because it sees more distinct values of the important parameter:
from scipy.stats import loguniform
from sklearn.model_selection import RandomizedSearchCV
rs = RandomizedSearchCV(pipe, {"svc__C": loguniform(1e-2, 1e2), "svc__gamma": loguniform(1e-3, 1e0)},
n_iter=20, cv=5, random_state=0)
rs.fit(X_train, y_train)loguniform produces uniform samples on a logarithmic scale.
Practice Random search
Run RandomizedSearchCV(..., n_iter=15, cv=5, random_state=0) on pipe with the distributions loguniform(1e-2, 1e2) (C) and loguniform(1e-3, 1e0) (gamma). Print best_score_ and the test score with round(..., 3) on one line.
Starter code
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
pipe = Pipeline([("olcek", StandardScaler()), ("svc", SVC())])
from scipy.stats import loguniform
from sklearn.model_selection import RandomizedSearchCV
# buraya yazSample solution
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
pipe = Pipeline([("olcek", StandardScaler()), ("svc", SVC())])
from scipy.stats import loguniform
from sklearn.model_selection import RandomizedSearchCV
rs = RandomizedSearchCV(pipe, {"svc__C": loguniform(1e-2, 1e2), "svc__gamma": loguniform(1e-3, 1e0)}, n_iter=15, cv=5, random_state=0)
rs.fit(X_train, y_train)
print(round(rs.best_score_, 3), round(rs.score(X_test, y_test), 3))Expected output
0.984 0.972
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 Why can a 1000-combination grid's best_score_ come out above 0.5 on randomly labelled data?
- Because the highest is picked by luck among many tries Correct answer
- Because the model found a real pattern
- A calculation error
- It can't
Explanation: Every combination's score is noisy; picking the largest creates an upward bias.
2 Try 30 random combinations.
rs = RandomizedSearchCV(pipe, dagilimlar, ____=30, cv=5, random_state=0)rs = RandomizedSearchCV(pipe, dagilimlar, n_iter=30, cv=5, random_state=0)Explanation: n_iter is the number of combinations to try.
3 Match the concepts.
| Term | Meaning |
|---|---|
best_score_ |
Optimistic selection score |
test skoru |
Honest measurement |
RandomizedSearchCV |
Random sampling |
loguniform |
Distribution on log scale |