What is cross-validation?
- Machine Learning course
- Intermediate
- Lesson 13/112
- 5 min
Cross-validation splits data into k folds and evaluates the model k times.
What does cross-validation solve?
The score of a single training/test split depends on which examples land in the test set. Cross-validation (CV) removes this luck: the data is split into folds and each fold becomes the validation set in turn. The result is a distribution of scores.
for rs in range(5):
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=rs)
print(rs, round(model.fit(X_tr, y_tr).score(X_te, y_te), 3))
# the score moves a few points from split to splitCV also uses the data efficiently: every example appears both in training and in validation.
Split-dependent variance
To measure split variance, the standard deviation of the scores is checked. With small datasets the variance is large and decisions based on a single split are misleading:
skorlar = cross_val_score(model, X, y, cv=10)
print(f"{skorlar.mean():.3f} ± {skorlar.std():.3f}")
print(skorlar.min(), skorlar.max()) # worst and best foldIf the difference between two models is smaller than the standard deviation of the scores, treating them as "equal" is safer.
Practice Split variance
Make test_size=0.2 splits with 5 different random_state values (0-4), train the model on each and collect the test accuracy. Print the smallest and largest score with round(..., 3) on one line.
Starter code
from sklearn.datasets import load_breast_cancer
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
X, y = load_breast_cancer(return_X_y=True)
model = make_pipeline(StandardScaler(), LogisticRegression())
from sklearn.model_selection import train_test_split
# buraya yazSample solution
from sklearn.datasets import load_breast_cancer
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
X, y = load_breast_cancer(return_X_y=True)
model = make_pipeline(StandardScaler(), LogisticRegression())
from sklearn.model_selection import train_test_split
skorlar = []
for rs in range(5):
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=rs)
skorlar.append(model.fit(X_tr, y_tr).score(X_te, y_te))
print(round(min(skorlar), 3), round(max(skorlar), 3))Expected output
0.965 0.974
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 Does cross-validation replace the test set?
- No; model selection with CV, the final measurement with a separate test set Correct answer
- Yes, always
- Only in regression
- The test set is unnecessary
Explanation: The CV score becomes optimistic as it is used for selection; an independent test is still needed.
2 Repeat the single-split score with different seeds.
for rs in ____(5):for rs in range(5):Explanation: range(5) produces seeds from 0 to 4.
3 Match the concepts.
| Term | Meaning |
|---|---|
fold |
A part of the data |
scores.std() |
Split variance |
test set |
Final independent measurement |
CV mean |
Selection criterion |