Metric selection
- Machine Learning course
- Intermediate
- Lesson 91/112
- 5 min
For imbalanced data, F1 or AUC is used instead of accuracy.
Choosing the metric
On imbalanced data, metrics focusing on the minority are used instead of accuracy:
- Precision/recall/F1: for the minority class.
- Average precision (PR AUC): the area under the precision-recall curve; threshold-independent.
- ROC AUC: can look optimistic under imbalance, because the many true negatives keep the FPR small.
from sklearn.metrics import average_precision_score, roc_auc_score
p = m.predict_proba(X_test)[:, 1]
print(roc_auc_score(y_test, p), average_precision_score(y_test, p))A random model's PR AUC equals the minority share (0.05 here); its ROC AUC is 0.5.
Threshold tuning
Without changing the model, the most effective step is often tuning the decision threshold. The threshold is chosen on validation data to give the desired recall or the highest F1:
from sklearn.metrics import precision_recall_curve
pr, re, esik = precision_recall_curve(y_val, p_val)
f1 = 2 * pr * re / (pr + re + 1e-12)
en_iyi = esik[f1[:-1].argmax()]TunedThresholdClassifierCV in newer scikit-learn versions does this automatically with CV.
Practice AUC and AP
Print the ROC AUC and the average precision of the logistic regression's test probabilities with round(..., 3) on one line. Then print the same two metrics for random probabilities (rng.random(len(y_test))).
Starter code
import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
X, y = make_classification(n_samples=2000, n_features=10, n_informative=4, weights=[0.95, 0.05], flip_y=0, random_state=1)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score, average_precision_score
rng = np.random.default_rng(0)
# buraya yazSample solution
import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
X, y = make_classification(n_samples=2000, n_features=10, n_informative=4, weights=[0.95, 0.05], flip_y=0, random_state=1)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score, average_precision_score
rng = np.random.default_rng(0)
p = LogisticRegression().fit(X_train, y_train).predict_proba(X_test)[:, 1]
print(round(roc_auc_score(y_test, p), 3), round(average_precision_score(y_test, p), 3))
r = rng.random(len(y_test))
print(round(roc_auc_score(y_test, r), 3), round(average_precision_score(y_test, r), 3))Expected output
0.928 0.425 0.455 0.046
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 With a 1% minority, roughly what is a random model's average precision?
- 0
- 0.01 Correct answer
- 0.5
- 0.99
Explanation: With random predictions, precision at every threshold is around the minority share.
2 Compute the threshold-independent PR summary.
ap = ____(y_test, p)ap = average_precision_score(y_test, p)Explanation: average_precision_score summarises the area under the PR curve.
3 Match the metrics.
| Term | Meaning |
|---|---|
average_precision |
Area under PR curve |
roc_auc |
Optimistic under imbalance |
F1 |
Threshold-dependent balance |
threshold tuning |
Improvement without changing the model |