Machine Learning
Advantages
Cross-validation gives more reliable results than a single train-test split.
Repeated CV
A single K-fold CV also depends on the fold assignment. RepeatedStratifiedKFold repeats CV a few times with different shuffles and gives a more stable mean:
from sklearn.model_selection import RepeatedStratifiedKFold
cv = RepeatedStratifiedKFold(n_splits=5, n_repeats=3, random_state=0)
skor = cross_val_score(model, X, y, cv=cv)
print(len(skor)) # 15
On small datasets it is one of the most reliable ways to judge the difference between two models.
Nested CV
When hyperparameters are chosen with CV, that CV score becomes optimistic. Nested CV measures with an outer CV while choosing with a separate inner CV; it gives an honest score for the entire selection process:
ic = GridSearchCV(SVC(), {"C": [0.1, 1, 10]}, cv=3)
dis_skor = cross_val_score(ic, X, y, cv=5) # a new search in every outer fold
print(dis_skor.mean())
It is expensive (5 × 3 × 3 trainings) but the most accurate method for a result to be published.
Exercise Nested validation
On Iris, evaluate the inner searcher GridSearchCV(SVC(), {"C": [0.1, 1, 10]}, cv=3) on the outside with cross_val_score(..., cv=5). Print the mean of the outer scores with round(..., 3), then the best_score_ the search finds on all the data with round(..., 3).
Show example solution
from sklearn.datasets import load_iris
from sklearn.model_selection import GridSearchCV, cross_val_score
from sklearn.svm import SVC
X, y = load_iris(return_X_y=True)
ic = GridSearchCV(SVC(), {"C": [0.1, 1, 10]}, cv=3)
print(round(cross_val_score(ic, X, y, cv=5).mean(), 3))
print(round(ic.fit(X, y).best_score_, 3))
Test yourself
-
How many trainings are done with <code>n_splits=10, n_repeats=5</code>?
Show answer
50 — 10 folds × 5 repeats = 50 trainings.
-
Repeat CV 3 times.
cv = RepeatedStratifiedKFold(n_splits=5, ____=3, random_state=0)
Show answer
cv = RepeatedStratifiedKFold(n_splits=5, n_repeats=3, random_state=0)
-
Match the methods.
- Repeated CV
- More stable mean
- Nested CV
- Honest score including selection
- Inner CV
- Hyperparameter choice
- Outer CV
- Generalisation measurement
Next lesson: Summary - Cross-validation
