Machine Learning
Recall
Recall shows how many true positives are found.
The ROC curve
As the threshold falls from 1 to 0, recall (the true positive rate, TPR) rises but so does the false positive rate (FPR). The ROC curve plots these two rates for all thresholds:
from sklearn.metrics import roc_curve
fpr, tpr, esikler = roc_curve(y_test, p)
# a (fpr, tpr) point for each threshold
FPR = FP / (FP + TN): the share of true negatives wrongly called positive.
AUC
The area under the ROC curve (AUC) is a one-number summary: 1.0 is perfect, 0.5 random. It is threshold-independent and is interpreted as "the probability that a random positive example scores higher than a random negative".
from sklearn.metrics import roc_auc_score
print(roc_auc_score([0, 0, 1, 1], [0.1, 0.4, 0.35, 0.8])) # 0.75
In three of the four positive-negative pairs the positive scored higher: 3/4 = 0.75.
Exercise Compute AUC
For the breast cancer test probabilities p, print the roc_auc_score with round(..., 3). Then print how many thresholds roc_curve produced (len(esikler)).
Show example solution
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
m = make_pipeline(StandardScaler(), LogisticRegression()).fit(X_train, y_train)
p = m.predict_proba(X_test)[:, 1]
from sklearn.metrics import roc_auc_score, roc_curve
print(round(roc_auc_score(y_test, p), 3))
fpr, tpr, esikler = roc_curve(y_test, p)
print(len(esikler))
Test yourself
-
What does the ROC curve of a model guessing randomly look like?
- The diagonal from (0,0) to (1,1)
- Hugging the top-left corner
- A horizontal line
- A single point
Show answer
The diagonal from (0,0) to (1,1) — For a random model TPR always equals FPR; its AUC is 0.5.
-
Compute the area under the ROC curve.
auc = ____(y_test, p)
Show answer
auc = roc_auc_score(y_test, p)
-
Match the terms.
- TPR
- Recall
- FPR
- FP / (FP + TN)
- AUC = 0.5
- Random
- AUC = 1.0
- Perfect ranking
Next lesson: F1 score
