Score comparison
- Machine Learning course
- Beginner
- Lesson 90/112
- 5 min
Models are compared using validation scores.
Comparing scores
A small difference between two models' CV means may not be meaningful. Besides the mean, the standard deviation is checked too; if the ranges overlap you can't say "one is definitely better":
for ad, m in adaylar.items():
s = cross_val_score(m, X, y, cv=10)
print(f"{ad}: {s.mean():.3f} ± {s.std():.3f}")If the scores are close, the simpler, faster or more interpretable model is preferred.
Training time is a criterion too
cross_validate returns training and prediction times alongside the scores, and can compute several metrics at once:
from sklearn.model_selection import cross_validate
r = cross_validate(m, X, y, cv=5, scoring=["accuracy", "f1_macro"])
print(r["test_accuracy"].mean(), r["test_f1_macro"].mean())
print(r["fit_time"].mean())If every prediction counts in milliseconds in a live system, a much slower model isn't chosen for a small accuracy difference.
Practice Mean ± deviation
Compare a scaled KNN and a scaled logistic regression with 5 folds on the breast cancer data. Print each line in the form f"{ad}: {ort:.3f} ± {std:.3f}".
Starter code
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
X, y = load_breast_cancer(return_X_y=True)
from sklearn.neighbors import KNeighborsClassifier
from sklearn.linear_model import LogisticRegression
adaylar = {"knn": make_pipeline(StandardScaler(), KNeighborsClassifier()), "lojistik": make_pipeline(StandardScaler(), LogisticRegression())}
# buraya yazSample solution
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
X, y = load_breast_cancer(return_X_y=True)
from sklearn.neighbors import KNeighborsClassifier
from sklearn.linear_model import LogisticRegression
adaylar = {"knn": make_pipeline(StandardScaler(), KNeighborsClassifier()), "lojistik": make_pipeline(StandardScaler(), LogisticRegression())}
for ad, m in adaylar.items():
s = cross_val_score(m, X, y, cv=5)
print(f"{ad}: {s.mean():.3f} ± {s.std():.3f}")Expected output
knn: 0.965 ± 0.010 lojistik: 0.981 ± 0.007
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 If A: 0.912 ± 0.040 and B: 0.918 ± 0.045, which interpretation is right?
- B is clearly better
- A is clearly better
- Both are bad
- The difference is within the uncertainty; practically equal Correct answer
Explanation: A difference of 0.006 is tiny next to standard deviations of ~0.04.
2 Validate with several metrics.
r = ____(m, X, y, cv=5, scoring=["accuracy", "f1_macro"])r = cross_validate(m, X, y, cv=5, scoring=["accuracy", "f1_macro"])Explanation: cross_validate returns a dict: test_*, fit_time, score_time.
3 Match the outputs.
| Term | Meaning |
|---|---|
s.mean() |
Expected success |
s.std() |
Uncertainty |
fit_time |
Training time |
test_f1_macro |
Macro F1 scores |