Machine Learning
Evaluation
Calculate model accuracy.
Evaluation
A single accuracy number isn't enough to evaluate the project. The confusion matrix shows which classes get mixed up, and classification_report shows precision and recall per class:
from sklearn.metrics import confusion_matrix, classification_report
tahmin = model.predict(X_test)
print(confusion_matrix(y_test, tahmin))
print(classification_report(y_test, tahmin, digits=3))
The class with the lowest recall in the report is the class the model struggles with most.
Per-class recall
recall_score(..., average=None) returns each class's recall as a separate array; it equals the diagonal of the confusion matrix divided by the row sums:
from sklearn.metrics import recall_score
print(recall_score(y_test, tahmin, average=None).round(3))
cm = confusion_matrix(y_test, tahmin)
print((cm.diagonal() / cm.sum(axis=1)).round(3)) # the same
Naming the weakest class when writing up results shows where the next improvement should focus.
Exercise Report the result
Print the confusion matrix with the test predictions. Then print the per-class recall with recall_score(..., average=None).round(3) and the index of the class with the lowest recall with argmin.
Show example solution
from sklearn.datasets import load_wine
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
X, y = load_wine(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42, stratify=y)
model = make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=5)).fit(X_train, y_train)
from sklearn.metrics import confusion_matrix, recall_score
tahmin = model.predict(X_test)
print(confusion_matrix(y_test, tahmin))
r = recall_score(y_test, tahmin, average=None).round(3)
print(r)
print(r.argmin())
Test yourself
-
In the confusion matrix, what does a 3 in row 2, column 1 mean?
- Correct predictions of class 1
- There are 3 classes
- Class 0 precision
- 3 examples truly of class 1 were predicted as class 0
Show answer
3 examples truly of class 1 were predicted as class 0 — The row shows the true class (index 1), the column the prediction (index 0).
-
Get each class's recall separately.
r = recall_score(y_test, tahmin, average=____)
Show answer
r = recall_score(y_test, tahmin, average=None)
-
Match the outputs.
- confusion_matrix
- Mixed-up classes
- classification_report
- Per-class table
- average=None
- Per-class array
- cm.diagonal()
- Correct predictions
Next lesson: What is scaling?
