Machine Learning
cross_val_score
The cross_val_score function returns k-fold cross-validation scores.
cross_val_score
cross_val_score does the whole loop you wrote by hand: splitting, cloning, training, measuring. cv takes a number or a splitter object; n_jobs=-1 runs the folds in parallel:
from sklearn.model_selection import cross_val_score
skorlar = cross_val_score(model, X, y, cv=5, scoring="roc_auc", n_jobs=-1)
print(skorlar.round(3), skorlar.mean().round(3))
With a classifier and a number, stratified splitting is used; with a regressor, plain KFold.
cross_validate and cross_val_predict
cross_validate returns several metrics, the times and, if requested, the training scores. cross_val_predict gives for every example the prediction of a model that hasn't seen it; ideal for a confusion matrix over all data:
from sklearn.model_selection import cross_validate, cross_val_predict
r = cross_validate(model, X, y, cv=5, scoring=["accuracy", "recall"], return_train_score=True)
print(r["train_accuracy"].mean(), r["test_accuracy"].mean())
p = cross_val_predict(model, X, y, cv=5)
The gap between training and validation scores is the sign of overfitting within CV.
Exercise Out-of-fold matrix
Get predictions for all the data with cross_val_predict(model, X, y, cv=5) and print the confusion matrix. Then use cross_validate with return_train_score=True and print the training and validation accuracy means with round(..., 3) on one line.
Show example solution
from sklearn.datasets import load_breast_cancer
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
X, y = load_breast_cancer(return_X_y=True)
model = make_pipeline(StandardScaler(), LogisticRegression())
from sklearn.model_selection import cross_val_predict, cross_validate
from sklearn.metrics import confusion_matrix
p = cross_val_predict(model, X, y, cv=5)
print(confusion_matrix(y, p))
r = cross_validate(model, X, y, cv=5, return_train_score=True)
print(round(r["train_score"].mean(), 3), round(r["test_score"].mean(), 3))
Test yourself
-
What does <code>n_jobs=-1</code> do?
- Makes the fold count unlimited
- Skips training
- Uses all processor cores in parallel
- Turns on debug mode
Show answer
Uses all processor cores in parallel — Each fold is independent, so parallelisation is straightforward.
-
Get, for every example, the prediction of a model that hasn't seen it.
p = ____(model, X, y, cv=5)
Show answer
p = cross_val_predict(model, X, y, cv=5)
-
Match the functions.
- cross_val_score
- Single-metric scores
- cross_validate
- Multiple metrics and times
- cross_val_predict
- Out-of-fold prediction
- return_train_score
- Training scores
Next lesson: Stratified
