Summary - Imbalanced data

In this group you learned SMOTE, class_weight, and metric selection.

Summary: imbalanced data

In this group:

Accuracy misleads on imbalanced data; minority recall and precision are checked.
SMOTE generates synthetic intermediate examples; random over/undersampling are simple alternatives; sampling only on training data.
class_weight makes minority errors costlier without changing the data.
Metrics: F1, average precision (random = minority share), ROC AUC (can be optimistic).
Threshold tuning is often the cheapest and most effective improvement.

Exercise The best F1 threshold

Split the training set into training and validation with test_size=0.3, random_state=1, stratify=y_train. Train logistic regression on training; on validation find the threshold maximising F1 with precision_recall_curve. Print the test F1 at the 0.5 threshold and at the found threshold with round(..., 3) on one line.

Show example solution

import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=2000, n_features=10, n_informative=4, weights=[0.95, 0.05], flip_y=0, random_state=1)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import f1_score, precision_recall_curve

X_tr, X_val, y_tr, y_val = train_test_split(X_train, y_train, test_size=0.3, random_state=1, stratify=y_train)
m = LogisticRegression().fit(X_tr, y_tr)
pr, re, esik = precision_recall_curve(y_val, m.predict_proba(X_val)[:, 1])
f1 = 2 * pr * re / (pr + re + 1e-12)
en_iyi = esik[f1[:-1].argmax()]
p = m.predict_proba(X_test)[:, 1]
print(round(f1_score(y_test, p >= 0.5), 3), round(f1_score(y_test, p >= en_iyi), 3))

Test yourself

How should CV be done on imbalanced data?
- With StratifiedKFold, sampling within each fold's training part
- SMOTE all the data first, then split
- With plain KFold
- CV isn't needed
Show answer

With StratifiedKFold, sampling within each fold's training part — Sampling first and then splitting leaks synthetic examples into the validation fold.

Make the selection by F1.

gs = GridSearchCV(pipe, izgara, cv=5, scoring=____)

Show answer

gs = GridSearchCV(pipe, izgara, cv=5, scoring="f1")

Match the remedies.

Data side

SMOTE, undersampling
Model side

class_weight
Decision side

Threshold tuning
Metric side

F1, average precision

Next lesson: Feature extraction

We Are Enhancing Your Experience

Cookie Preferences

Strictly Necessary Cookies

Analytical Cookies