cross_val_score
- Machine Learning course
- Intermediate
- Lesson 15/112
- 5 min
The cross_val_score function returns k-fold cross-validation scores.
cross_val_score
cross_val_score does the whole loop you wrote by hand: splitting, cloning, training, measuring. cv takes a number or a splitter object; n_jobs=-1 runs the folds in parallel:
from sklearn.model_selection import cross_val_score
skorlar = cross_val_score(model, X, y, cv=5, scoring="roc_auc", n_jobs=-1)
print(skorlar.round(3), skorlar.mean().round(3))With a classifier and a number, stratified splitting is used; with a regressor, plain KFold.
cross_validate and cross_val_predict
cross_validate returns several metrics, the times and, if requested, the training scores. cross_val_predict gives for every example the prediction of a model that hasn't seen it; ideal for a confusion matrix over all data:
from sklearn.model_selection import cross_validate, cross_val_predict
r = cross_validate(model, X, y, cv=5, scoring=["accuracy", "recall"], return_train_score=True)
print(r["train_accuracy"].mean(), r["test_accuracy"].mean())
p = cross_val_predict(model, X, y, cv=5)The gap between training and validation scores is the sign of overfitting within CV.
Practice Out-of-fold matrix
Get predictions for all the data with cross_val_predict(model, X, y, cv=5) and print the confusion matrix. Then use cross_validate with return_train_score=True and print the training and validation accuracy means with round(..., 3) on one line.
Starter code
from sklearn.datasets import load_breast_cancer
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
X, y = load_breast_cancer(return_X_y=True)
model = make_pipeline(StandardScaler(), LogisticRegression())
from sklearn.model_selection import cross_val_predict, cross_validate
from sklearn.metrics import confusion_matrix
# buraya yazSample solution
from sklearn.datasets import load_breast_cancer
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
X, y = load_breast_cancer(return_X_y=True)
model = make_pipeline(StandardScaler(), LogisticRegression())
from sklearn.model_selection import cross_val_predict, cross_validate
from sklearn.metrics import confusion_matrix
p = cross_val_predict(model, X, y, cv=5)
print(confusion_matrix(y, p))
r = cross_validate(model, X, y, cv=5, return_train_score=True)
print(round(r["train_score"].mean(), 3), round(r["test_score"].mean(), 3))Expected output
[[204 8] [ 3 354]] 0.989 0.981
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 What does <code>n_jobs=-1</code> do?
- Makes the fold count unlimited
- Skips training
- Uses all processor cores in parallel Correct answer
- Turns on debug mode
Explanation: Each fold is independent, so parallelisation is straightforward.
2 Get, for every example, the prediction of a model that hasn't seen it.
p = ____(model, X, y, cv=5)p = cross_val_predict(model, X, y, cv=5)Explanation: cross_val_predict produces out-of-fold predictions.
3 Match the functions.
| Term | Meaning |
|---|---|
cross_val_score |
Single-metric scores |
cross_validate |
Multiple metrics and times |
cross_val_predict |
Out-of-fold prediction |
return_train_score |
Training scores |