Machine Learning
Accuracy
Accuracy is the ratio of correct predictions to total predictions.
Accuracy
Accuracy is the share of correct predictions among all predictions. The score method of classifiers returns it:
from sklearn.metrics import accuracy_score
y_true = [1, 0, 1, 1, 0]
y_pred = [1, 0, 0, 1, 0]
print(accuracy_score(y_true, y_pred)) # 0.8
print((np.array(y_true) == np.array(y_pred)).mean()) # the same
It is the easiest metric to understand and gives a good summary when the classes are balanced.
The trap with imbalanced classes
In data where 1% of transactions are fraud, a model saying "everything is normal" gets 99% accuracy but catches no fraud at all:
y_true = np.array([0] * 99 + [1])
y_pred = np.zeros(100, dtype=int)
print(accuracy_score(y_true, y_pred)) # 0.99, yet useless
That's why with imbalanced problems the confusion matrix, precision and recall are examined alongside accuracy.
Exercise Compare with a baseline
Build a baseline that always predicts the most frequent class with DummyClassifier(strategy="most_frequent"). Print the test accuracy of both it and a LogisticRegression(max_iter=5000) model with round(..., 3) on separate lines.
Show example solution
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
temel = DummyClassifier(strategy="most_frequent").fit(X_train, y_train)
m = LogisticRegression(max_iter=5000).fit(X_train, y_train)
print(round(temel.score(X_test, y_test), 3))
print(round(m.score(X_test, y_test), 3))
Test yourself
-
If 170 of 200 predictions are correct, what is the accuracy?
Show answer
0.85 — 170 / 200 = 0.85.
-
Compute the accuracy with the metric function.
acc = ____(y_test, y_pred)
Show answer
acc = accuracy_score(y_test, y_pred)
-
Match the situations.
- Balanced classes
- Accuracy is meaningful
- 99% one class
- Accuracy is misleading
- m.score(X, y)
- Accuracy for a classifier
- y_true == y_pred
- Correct-prediction mask
Next lesson: Confusion matrix
