Machine Learning
K-fold
K-fold splits data into k equal parts and tests each part once.
KFold
KFold splits the data into K consecutive parts. If the data has an order (all 0s first, then all 1s), splitting without shuffling makes the folds unbalanced; that is why shuffle=True is often used:
from sklearn.model_selection import KFold
kf = KFold(n_splits=5, shuffle=True, random_state=0)
for egitim, dogrulama in kf.split(X):
print(len(egitim), len(dogrulama))
split produces index arrays; the data isn't copied.
Choosing K
As K grows, each training set approaches the full data (a less pessimistic estimate), but the number of trainings and the time grow. Common choices are 5 and 10. The extreme case is leave-one-out: K = number of examples.
from sklearn.model_selection import LeaveOneOut
print(LeaveOneOut().get_n_splits(X)) # as many trainings as examples
On large datasets even 3 folds may be enough; on small ones 10 folds or repeated CV are preferred.
Exercise A CV loop by hand
Walk through the folds with KFold(n_splits=5, shuffle=True, random_state=0). In each fold train clone(model) with the training indices and collect the validation accuracy. Print the mean with round(..., 3).
Show example solution
import numpy as np
from sklearn.base import clone
from sklearn.model_selection import KFold
from sklearn.datasets import load_breast_cancer
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
X, y = load_breast_cancer(return_X_y=True)
model = make_pipeline(StandardScaler(), LogisticRegression())
skor = []
for egitim, dogrulama in KFold(n_splits=5, shuffle=True, random_state=0).split(X):
m = clone(model).fit(X[egitim], y[egitim])
skor.append(m.score(X[dogrulama], y[dogrulama]))
print(round(np.mean(skor), 3))
Test yourself
-
With 100 examples and <code>n_splits=4</code>, how many examples are in each validation fold?
Show answer
25 — 100 / 4 = 25 for validation, the remaining 75 for training.
-
Shuffle before splitting into folds.
kf = KFold(n_splits=5, ____=True, random_state=0)
Show answer
kf = KFold(n_splits=5, shuffle=True, random_state=0)
-
Match the splitters.
- KFold
- K consecutive parts
- LeaveOneOut
- One example each time
- shuffle=True
- Shuffles first
- split(X)
- Produces indices
Next lesson: cross_val_score
