Machine Learning
The score method
The score() method returns the model's R² or accuracy value.
The score method
score(X, y) returns the default metric for each estimator type: accuracy for classifiers, R² for regressors. If a different metric is needed, the sklearn.metrics functions or get_scorer are used:
from sklearn.metrics import get_scorer
f1 = get_scorer("f1_macro")
print(f1(m, X_test, y_test)) # (model, X, y) signature
Scorer objects take (estimator, X, y); this is the form cross-validation and grid search use internally.
make_scorer
You can turn your own metric into a scorer with make_scorer. If the error metric is better when smaller, greater_is_better=False is given; the scorer returns a negative result:
from sklearn.metrics import make_scorer
def ort_yuzde_hata(y, p):
return np.mean(np.abs(y - p) / y)
olcut = make_scorer(ort_yuzde_hata, greater_is_better=False)
cross_val_score(model, X, y, cv=5, scoring=olcut)
Business-specific costs (like a false negative being 10 times costlier) enter model selection directly this way.
Exercise Custom scorer
Write a maliyet(y, p) function counting a false negative as 5 points and a false positive as 1 point. Make a scorer with make_scorer(maliyet, greater_is_better=False) and print the model's value on the test set.
Show example solution
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import make_scorer
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
m = make_pipeline(StandardScaler(), LogisticRegression()).fit(X_train, y_train)
def maliyet(y, p):
fn = np.sum((y == 1) & (p == 0))
fp = np.sum((y == 0) & (p == 1))
return 5 * fn + fp
olcut = make_scorer(maliyet, greater_is_better=False)
print(olcut(m, X_test, y_test))
Test yourself
-
What does <code>RandomForestRegressor().score(X, y)</code> return?
Show answer
R² — The default score metric of every regressor is R².
-
Make a scorer from a smaller-is-better metric.
olcut = make_scorer(hata, ____=False)
Show answer
olcut = make_scorer(hata, greater_is_better=False)
-
Match the default metrics.
- Classifier score
- Accuracy
- Regressor score
- R²
- get_scorer
- Scorer by name
- make_scorer
- Custom scorer
Next lesson: Summary - scikit-learn
