Evaluation
- Machine Learning course
- Intermediate
- Lesson 34/112
- 7 min
Calculate SVM accuracy.
Evaluation
The final measurement is done once on the test set: F1, precision, recall and the confusion matrix. The result is reported in comparison with the baseline:
p = gs.predict(X_test)
print(classification_report(y_test, p, digits=3))
print(confusion_matrix(y_test, p))The report should include: which metric was chosen and why, the gain over the baseline, and how many of each error were made.
Telling the result
Numbers must be translated into business language: "We caught X of the 31 leaving customers in the test set; Y of every 10 customers we flagged really left." These sentences are recall and precision themselves:
tn, fp, fn, tp = confusion_matrix(y_test, p).ravel()
print(f"{tp + fn} ayrılandan {tp} yakalandı")
print(f"{tp + fp} alarmın {tp} tanesi doğru")Limitations must also be written: the data is synthetic, the sample is small, the results must be revalidated on a real customer base.
Practice Final report
Get the test predictions with the given gs. Print the F1 with round(..., 3). Then unpack the confusion matrix with ravel() and print the lines f"{tp + fn} ayrılandan {tp} yakalandı" and f"{tp + fp} alarmın {tp} tanesi doğru".
Starter code
import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
X, y = make_classification(n_samples=600, n_features=8, n_informative=4, weights=[0.8, 0.2], random_state=7)
rng = np.random.default_rng(7)
X[rng.random(X.shape) < 0.05] = np.nan
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
from sklearn.pipeline import make_pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
from sklearn.model_selection import GridSearchCV
pipe = make_pipeline(SimpleImputer(strategy="median"), StandardScaler(), SVC(class_weight="balanced"))
gs = GridSearchCV(pipe, {"svc__C": [0.1, 1, 10], "svc__gamma": ["scale", 0.1]}, cv=5, scoring="f1").fit(X_train, y_train)
from sklearn.metrics import f1_score, confusion_matrix
# buraya yazSample solution
import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
X, y = make_classification(n_samples=600, n_features=8, n_informative=4, weights=[0.8, 0.2], random_state=7)
rng = np.random.default_rng(7)
X[rng.random(X.shape) < 0.05] = np.nan
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
from sklearn.pipeline import make_pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
from sklearn.model_selection import GridSearchCV
pipe = make_pipeline(SimpleImputer(strategy="median"), StandardScaler(), SVC(class_weight="balanced"))
gs = GridSearchCV(pipe, {"svc__C": [0.1, 1, 10], "svc__gamma": ["scale", 0.1]}, cv=5, scoring="f1").fit(X_train, y_train)
from sklearn.metrics import f1_score, confusion_matrix
p = gs.predict(X_test)
print(round(f1_score(y_test, p), 3))
tn, fp, fn, tp = confusion_matrix(y_test, p).ravel()
print(f"{tp + fn} ayrılandan {tp} yakalandı")
print(f"{tp + fp} alarmın {tp} tanesi doğru")Expected output
0.833 31 ayrılandan 25 yakalandı 29 alarmın 25 tanesi doğru
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 The test F1 came out slightly lower than <code>best_score_</code> in CV. What does that mean?
- The model is useless
- It is expected; best_score_ is optimistic Correct answer
- There is definitely a bug
- The test set is broken
Explanation: A small drop is normal because of the luck involved in the selection process.
2 Get the per-class report with three digits.
print(____(y_test, p, digits=3))print(classification_report(y_test, p, digits=3))Explanation: classification_report gives precision, recall, F1 and support.
3 Match the sentences with the metrics.
| Term | Meaning |
|---|---|
How many of those who left did we catch? |
Recall |
How many alarms are right? |
Precision |
Balance of the two |
F1 |
Error breakdown |
Confusion matrix |