Machine Learning
What is imbalanced data?
Imbalanced data has classes with unequal numbers of examples.
What is imbalanced data?
If one class has far fewer examples than the others, the data is imbalanced: fraudulent transactions (0.1%), rare diseases (1%), machine failures. A model gets high accuracy by predicting the majority class but misses the minority that is actually sought.
print(np.bincount(y)) # [1900 100]
print(np.bincount(y) / len(y)) # [0.95 0.05]
Imbalance isn't an error but the nature of the problem; the fix is the right metric and the right training strategy.
The effect of imbalance
Because most models try to minimise the total error, they disregard the minority's errors. The result: the minority class's recall drops and the decision boundary shifts into the minority.
m = LogisticRegression().fit(X_train, y_train)
p = m.predict(X_test)
print(accuracy_score(y_test, p)) # high
print(recall_score(y_test, p)) # minority: low
In this group you'll see the remedies: resampling, class weights, threshold tuning and the right metric.
Exercise See the problem
Print the class distribution. Train a LogisticRegression(); print the test accuracy and the minority recall with round(..., 3) on one line.
Show example solution
import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
X, y = make_classification(n_samples=2000, n_features=10, n_informative=4, weights=[0.95, 0.05], flip_y=0, random_state=1)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, recall_score
print(np.bincount(y))
m = LogisticRegression().fit(X_train, y_train)
p = m.predict(X_test)
print(round(accuracy_score(y_test, p), 3), round(recall_score(y_test, p), 3))
Test yourself
-
With a 5% minority, what is the accuracy of a model that says "majority" to everything?
Show answer
0.95 — Accuracy looks high but the minority recall is 0.
-
Compute the class proportions.
print(np.bincount(y) / ____)
Show answer
print(np.bincount(y) / len(y))
-
Match the examples.
- fraud
- Very rare minority
- majority prediction
- High accuracy, zero recall
- stratify
- Keeps the proportion
- recall
- Catching the minority
Next lesson: SMOTE
