Machine Learning
What is cross-validation?
Cross-validation splits data into k folds and evaluates the model k times.
What does cross-validation solve?
The score of a single training/test split depends on which examples land in the test set. Cross-validation (CV) removes this luck: the data is split into folds and each fold becomes the validation set in turn. The result is a distribution of scores.
for rs in range(5):
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=rs)
print(rs, round(model.fit(X_tr, y_tr).score(X_te, y_te), 3))
# the score moves a few points from split to split
CV also uses the data efficiently: every example appears both in training and in validation.
Split-dependent variance
To measure split variance, the standard deviation of the scores is checked. With small datasets the variance is large and decisions based on a single split are misleading:
skorlar = cross_val_score(model, X, y, cv=10)
print(f"{skorlar.mean():.3f} ± {skorlar.std():.3f}")
print(skorlar.min(), skorlar.max()) # worst and best fold
If the difference between two models is smaller than the standard deviation of the scores, treating them as "equal" is safer.
Exercise Split variance
Make test_size=0.2 splits with 5 different random_state values (0-4), train the model on each and collect the test accuracy. Print the smallest and largest score with round(..., 3) on one line.
Show example solution
from sklearn.datasets import load_breast_cancer
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
X, y = load_breast_cancer(return_X_y=True)
model = make_pipeline(StandardScaler(), LogisticRegression())
from sklearn.model_selection import train_test_split
skorlar = []
for rs in range(5):
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=rs)
skorlar.append(model.fit(X_tr, y_tr).score(X_te, y_te))
print(round(min(skorlar), 3), round(max(skorlar), 3))
Test yourself
-
Does cross-validation replace the test set?
- No; model selection with CV, the final measurement with a separate test set
- Yes, always
- Only in regression
- The test set is unnecessary
Show answer
No; model selection with CV, the final measurement with a separate test set — The CV score becomes optimistic as it is used for selection; an independent test is still needed.
-
Repeat the single-split score with different seeds.
for rs in ____(5):
Show answer
for rs in range(5):
-
Match the concepts.
- fold
- A part of the data
- scores.std()
- Split variance
- test set
- Final independent measurement
- CV mean
- Selection criterion
Next lesson: K-fold
