Advantages
- Machine Learning course
- Intermediate
- Lesson 17/112
- 5 min
Cross-validation gives more reliable results than a single train-test split.
Repeated CV
A single K-fold CV also depends on the fold assignment. RepeatedStratifiedKFold repeats CV a few times with different shuffles and gives a more stable mean:
from sklearn.model_selection import RepeatedStratifiedKFold
cv = RepeatedStratifiedKFold(n_splits=5, n_repeats=3, random_state=0)
skor = cross_val_score(model, X, y, cv=cv)
print(len(skor)) # 15On small datasets it is one of the most reliable ways to judge the difference between two models.
Nested CV
When hyperparameters are chosen with CV, that CV score becomes optimistic. Nested CV measures with an outer CV while choosing with a separate inner CV; it gives an honest score for the entire selection process:
ic = GridSearchCV(SVC(), {"C": [0.1, 1, 10]}, cv=3)
dis_skor = cross_val_score(ic, X, y, cv=5) # a new search in every outer fold
print(dis_skor.mean())It is expensive (5 × 3 × 3 trainings) but the most accurate method for a result to be published.
Practice Nested validation
On Iris, evaluate the inner searcher GridSearchCV(SVC(), {"C": [0.1, 1, 10]}, cv=3) on the outside with cross_val_score(..., cv=5). Print the mean of the outer scores with round(..., 3), then the best_score_ the search finds on all the data with round(..., 3).
Starter code
from sklearn.datasets import load_iris
from sklearn.model_selection import GridSearchCV, cross_val_score
from sklearn.svm import SVC
X, y = load_iris(return_X_y=True)
# buraya yazSample solution
from sklearn.datasets import load_iris
from sklearn.model_selection import GridSearchCV, cross_val_score
from sklearn.svm import SVC
X, y = load_iris(return_X_y=True)
ic = GridSearchCV(SVC(), {"C": [0.1, 1, 10]}, cv=3)
print(round(cross_val_score(ic, X, y, cv=5).mean(), 3))
print(round(ic.fit(X, y).best_score_, 3))Expected output
0.98 0.973
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 How many trainings are done with <code>n_splits=10, n_repeats=5</code>?
- 50 Correct answer
- 15
- 10
- 5
Explanation: 10 folds × 5 repeats = 50 trainings.
2 Repeat CV 3 times.
cv = RepeatedStratifiedKFold(n_splits=5, ____=3, random_state=0)cv = RepeatedStratifiedKFold(n_splits=5, n_repeats=3, random_state=0)Explanation: n_repeats is the number of repetitions.
3 Match the methods.
| Term | Meaning |
|---|---|
Repeated CV |
More stable mean |
Nested CV |
Honest score including selection |
Inner CV |
Hyperparameter choice |
Outer CV |
Generalisation measurement |