Machine Learning
Score comparison
Models are compared using validation scores.
Comparing scores
A small difference between two models' CV means may not be meaningful. Besides the mean, the standard deviation is checked too; if the ranges overlap you can't say "one is definitely better":
for ad, m in adaylar.items():
s = cross_val_score(m, X, y, cv=10)
print(f"{ad}: {s.mean():.3f} ± {s.std():.3f}")
If the scores are close, the simpler, faster or more interpretable model is preferred.
Training time is a criterion too
cross_validate returns training and prediction times alongside the scores, and can compute several metrics at once:
from sklearn.model_selection import cross_validate
r = cross_validate(m, X, y, cv=5, scoring=["accuracy", "f1_macro"])
print(r["test_accuracy"].mean(), r["test_f1_macro"].mean())
print(r["fit_time"].mean())
If every prediction counts in milliseconds in a live system, a much slower model isn't chosen for a small accuracy difference.
Exercise Mean ± deviation
Compare a scaled KNN and a scaled logistic regression with 5 folds on the breast cancer data. Print each line in the form f"{ad}: {ort:.3f} ± {std:.3f}".
Show example solution
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
X, y = load_breast_cancer(return_X_y=True)
from sklearn.neighbors import KNeighborsClassifier
from sklearn.linear_model import LogisticRegression
adaylar = {"knn": make_pipeline(StandardScaler(), KNeighborsClassifier()), "lojistik": make_pipeline(StandardScaler(), LogisticRegression())}
for ad, m in adaylar.items():
s = cross_val_score(m, X, y, cv=5)
print(f"{ad}: {s.mean():.3f} ± {s.std():.3f}")
Test yourself
-
If A: 0.912 ± 0.040 and B: 0.918 ± 0.045, which interpretation is right?
- B is clearly better
- A is clearly better
- Both are bad
- The difference is within the uncertainty; practically equal
Show answer
The difference is within the uncertainty; practically equal — A difference of 0.006 is tiny next to standard deviations of ~0.04.
-
Validate with several metrics.
r = ____(m, X, y, cv=5, scoring=["accuracy", "f1_macro"])
Show answer
r = cross_validate(m, X, y, cv=5, scoring=["accuracy", "f1_macro"])
-
Match the outputs.
- s.mean()
- Expected success
- s.std()
- Uncertainty
- fit_time
- Training time
- test_f1_macro
- Macro F1 scores
Next lesson: Hyperparameter
