Cross-validation

Cross-validation evaluates the model by splitting data into multiple folds.

Cross-validation

A single split depends on luck. K-fold cross-validation splits the data into K parts; each time one part is for validation and the rest for training. The mean of the K scores is a more reliable estimate:

from sklearn.model_selection import cross_val_score
skorlar = cross_val_score(KNeighborsClassifier(), X, y, cv=5)
print(skorlar.round(3))          # 5 separate scores
print(skorlar.mean().round(3), skorlar.std().round(3))

The standard deviation shows how much the score moves from split to split.

Stratified folds for classification

When a classifier is given, cross_val_score uses StratifiedKFold by default; class proportions are kept in each fold. The metric is chosen with scoring:

cross_val_score(model, X, y, cv=5, scoring="f1_macro")
cross_val_score(reg, X, y, cv=5, scoring="neg_mean_absolute_error")

Error metrics are given negative with "neg_", because scikit-learn always applies the "bigger is better" rule.

Exercise 5-fold validation

Run 5-fold cross-validation on Iris for a DecisionTreeClassifier(random_state=0). Print the scores with round(3), then the mean with round(..., 3).

Show example solution

from sklearn.datasets import load_iris
from sklearn.model_selection import cross_val_score

X, y = load_iris(return_X_y=True)
from sklearn.tree import DecisionTreeClassifier

skorlar = cross_val_score(DecisionTreeClassifier(random_state=0), X, y, cv=5)
print(skorlar.round(3))
print(round(skorlar.mean(), 3))

Test yourself

With <code>cv=10</code>, how many times is each example in the validation set?
- 0
- 1
- 10
- 9
Show answer

1 — Each example belongs to exactly one fold; it is measured once when that fold is the validation set.

Use macro F1 as the metric.

cross_val_score(model, X, y, cv=5, ____="f1_macro")

Show answer

cross_val_score(model, X, y, cv=5, scoring="f1_macro")

Match the concepts.

cv=5

5 folds
skorlar.mean()

Mean success
scores.std()

Split instability
neg_mean_absolute_error

Negative MAE

Next lesson: Grid search

We Are Enhancing Your Experience

Cookie Preferences

Strictly Necessary Cookies

Analytical Cookies