Machine Learning
best_score_
The best_score_ property gives the best cross-validation score.
Why is best_score_ optimistic?
best_score_ is the CV mean of the best combination. Since the highest of many combinations is picked, it tends to be a lucky result; the more combinations tried, the more optimism.
print(gs.best_score_) # the score of the selection: optimistic
print(gs.score(X_test, y_test)) # independent score: honest
The success to report is the test score or a nested CV score; best_score_ is only for ranking candidates.
RandomizedSearchCV
As the grid grows, the time explodes. RandomizedSearchCV tries n_iter random combinations from distributions; with the same budget it often does as well as a grid, because it sees more distinct values of the important parameter:
from scipy.stats import loguniform
from sklearn.model_selection import RandomizedSearchCV
rs = RandomizedSearchCV(pipe, {"svc__C": loguniform(1e-2, 1e2), "svc__gamma": loguniform(1e-3, 1e0)},
n_iter=20, cv=5, random_state=0)
rs.fit(X_train, y_train)
loguniform produces uniform samples on a logarithmic scale.
Exercise Random search
Run RandomizedSearchCV(..., n_iter=15, cv=5, random_state=0) on pipe with the distributions loguniform(1e-2, 1e2) (C) and loguniform(1e-3, 1e0) (gamma). Print best_score_ and the test score with round(..., 3) on one line.
Show example solution
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
pipe = Pipeline([("olcek", StandardScaler()), ("svc", SVC())])
from scipy.stats import loguniform
from sklearn.model_selection import RandomizedSearchCV
rs = RandomizedSearchCV(pipe, {"svc__C": loguniform(1e-2, 1e2), "svc__gamma": loguniform(1e-3, 1e0)}, n_iter=15, cv=5, random_state=0)
rs.fit(X_train, y_train)
print(round(rs.best_score_, 3), round(rs.score(X_test, y_test), 3))
Test yourself
-
Why can a 1000-combination grid's best_score_ come out above 0.5 on randomly labelled data?
- Because the highest is picked by luck among many tries
- Because the model found a real pattern
- A calculation error
- It can't
Show answer
Because the highest is picked by luck among many tries — Every combination's score is noisy; picking the largest creates an upward bias.
-
Try 30 random combinations.
rs = RandomizedSearchCV(pipe, dagilimlar, ____=30, cv=5, random_state=0)
Show answer
rs = RandomizedSearchCV(pipe, dagilimlar, n_iter=30, cv=5, random_state=0)
-
Match the concepts.
- best_score_
- Optimistic selection score
- test skoru
- Honest measurement
- RandomizedSearchCV
- Random sampling
- loguniform
- Distribution on log scale
Next lesson: Summary - Grid search
