Machine Learning
Metric selection
For imbalanced data, F1 or AUC is used instead of accuracy.
Choosing the metric
On imbalanced data, metrics focusing on the minority are used instead of accuracy:
- Precision/recall/F1: for the minority class.
- Average precision (PR AUC): the area under the precision-recall curve; threshold-independent.
- ROC AUC: can look optimistic under imbalance, because the many true negatives keep the FPR small.
from sklearn.metrics import average_precision_score, roc_auc_score
p = m.predict_proba(X_test)[:, 1]
print(roc_auc_score(y_test, p), average_precision_score(y_test, p))
A random model's PR AUC equals the minority share (0.05 here); its ROC AUC is 0.5.
Threshold tuning
Without changing the model, the most effective step is often tuning the decision threshold. The threshold is chosen on validation data to give the desired recall or the highest F1:
from sklearn.metrics import precision_recall_curve
pr, re, esik = precision_recall_curve(y_val, p_val)
f1 = 2 * pr * re / (pr + re + 1e-12)
en_iyi = esik[f1[:-1].argmax()]
TunedThresholdClassifierCV in newer scikit-learn versions does this automatically with CV.
Exercise AUC and AP
Print the ROC AUC and the average precision of the logistic regression's test probabilities with round(..., 3) on one line. Then print the same two metrics for random probabilities (rng.random(len(y_test))).
Show example solution
import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
X, y = make_classification(n_samples=2000, n_features=10, n_informative=4, weights=[0.95, 0.05], flip_y=0, random_state=1)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score, average_precision_score
rng = np.random.default_rng(0)
p = LogisticRegression().fit(X_train, y_train).predict_proba(X_test)[:, 1]
print(round(roc_auc_score(y_test, p), 3), round(average_precision_score(y_test, p), 3))
r = rng.random(len(y_test))
print(round(roc_auc_score(y_test, r), 3), round(average_precision_score(y_test, r), 3))
Test yourself
-
With a 1% minority, roughly what is a random model's average precision?
Show answer
0.01 — With random predictions, precision at every threshold is around the minority share.
-
Compute the threshold-independent PR summary.
ap = ____(y_test, p)
Show answer
ap = average_precision_score(y_test, p)
-
Match the metrics.
- average_precision
- Area under PR curve
- roc_auc
- Optimistic under imbalance
- F1
- Threshold-dependent balance
- threshold tuning
- Improvement without changing the model
Next lesson: Summary - Imbalanced data
