Summary - Imbalanced data
- Machine Learning course
- Intermediate
- Lesson 92/112
- 6 min
In this group you learned SMOTE, class_weight, and metric selection.
Summary: imbalanced data
In this group:
- Accuracy misleads on imbalanced data; minority recall and precision are checked.
- SMOTE generates synthetic intermediate examples; random over/undersampling are simple alternatives; sampling only on training data.
class_weightmakes minority errors costlier without changing the data.- Metrics: F1, average precision (random = minority share), ROC AUC (can be optimistic).
- Threshold tuning is often the cheapest and most effective improvement.
Practice The best F1 threshold
Split the training set into training and validation with test_size=0.3, random_state=1, stratify=y_train. Train logistic regression on training; on validation find the threshold maximising F1 with precision_recall_curve. Print the test F1 at the 0.5 threshold and at the found threshold with round(..., 3) on one line.
Starter code
import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
X, y = make_classification(n_samples=2000, n_features=10, n_informative=4, weights=[0.95, 0.05], flip_y=0, random_state=1)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import f1_score, precision_recall_curve
# buraya yazSample solution
import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
X, y = make_classification(n_samples=2000, n_features=10, n_informative=4, weights=[0.95, 0.05], flip_y=0, random_state=1)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import f1_score, precision_recall_curve
X_tr, X_val, y_tr, y_val = train_test_split(X_train, y_train, test_size=0.3, random_state=1, stratify=y_train)
m = LogisticRegression().fit(X_tr, y_tr)
pr, re, esik = precision_recall_curve(y_val, m.predict_proba(X_val)[:, 1])
f1 = 2 * pr * re / (pr + re + 1e-12)
en_iyi = esik[f1[:-1].argmax()]
p = m.predict_proba(X_test)[:, 1]
print(round(f1_score(y_test, p >= 0.5), 3), round(f1_score(y_test, p >= en_iyi), 3))Expected output
0.061 0.162
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 How should CV be done on imbalanced data?
- With StratifiedKFold, sampling within each fold's training part Correct answer
- SMOTE all the data first, then split
- With plain KFold
- CV isn't needed
Explanation: Sampling first and then splitting leaks synthetic examples into the validation fold.
2 Make the selection by F1.
Question
gs = GridSearchCV(pipe, izgara, cv=5, scoring=____)
Answer
gs = GridSearchCV(pipe, izgara, cv=5, scoring="f1")Explanation: In an imbalanced problem, scoring must take the minority into account.
3 Match the remedies.
| Term | Meaning |
|---|---|
Data side |
SMOTE, undersampling |
Model side |
class_weight |
Decision side |
Threshold tuning |
Metric side |
F1, average precision |