Machine Learning
Stratified
Stratified k-fold preserves class distribution in each fold.
StratifiedKFold
In classification the folds must keep the class proportions; otherwise one fold might have no minority class at all. StratifiedKFold guarantees this:
from sklearn.model_selection import StratifiedKFold
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
for egitim, dogrulama in skf.split(X, y): # y is needed
print(np.bincount(y[dogrulama]))
This time split also takes y, because it keeps the proportions by looking at it.
Grouped and time-based splitting
If several measurements of the same patient are in both training and validation, the model "recognises" the patient and the score inflates. GroupKFold keeps all examples of a group in the same fold. For time series, TimeSeriesSplit always trains on the past and validates on the future:
from sklearn.model_selection import GroupKFold, TimeSeriesSplit
cross_val_score(model, X, y, cv=GroupKFold(5), groups=hasta_id)
for tr, va in TimeSeriesSplit(n_splits=3).split(X):
print(tr.max() < va.min()) # True: training always comes first
The right splitter is the one that mimics how the data will really be used.
Exercise Fold proportions
For y made of 12 zeros and 3 ones, compute the number of 1s in the folds of KFold(n_splits=3) and StratifiedKFold(n_splits=3) (both without shuffling) as a list with y[dogrulama].sum(). Print the two lists on separate lines.
Show example solution
import numpy as np
from sklearn.model_selection import KFold, StratifiedKFold
X = np.arange(15).reshape(-1, 1)
y = np.array([0] * 12 + [1] * 3)
for bolucu in [KFold(n_splits=3), StratifiedKFold(n_splits=3)]:
print([int(y[dogrulama].sum()) for _, dogrulama in bolucu.split(X, y)])
Test yourself
-
Why does <code>StratifiedKFold.split</code> need the <code>y</code> parameter?
- It doesn't
- To keep the class proportions
- To train the model
- To compute scores
Show answer
To keep the class proportions — Distribution into folds is done for each class separately.
-
Keep the examples of the same group in the same fold.
cross_val_score(model, X, y, cv=GroupKFold(5), ____=hasta_id)
Show answer
cross_val_score(model, X, y, cv=GroupKFold(5), groups=hasta_id)
-
Match the situations with the splitters.
- Imbalanced classes
- StratifiedKFold
- Measurements of the same patient
- GroupKFold
- Daily sales
- TimeSeriesSplit
- Independent samples
- KFold
Next lesson: Advantages
