Challenge 2
- Machine Learning course
- Intermediate
- Lesson 110/112
- 8 min
Evaluate a model with cross-validation.
Challenge 2: honest evaluation
The honest score of a model selection process is measured with nested CV. In this challenge you'll compare three model families (logistic, random forest, gradient boosting) in an outer CV, each tuned in an inner CV with its own small grid:
ic = GridSearchCV(RandomForestClassifier(random_state=0),
{"max_depth": [None, 6]}, cv=3)
dis = cross_val_score(ic, X, y, cv=5) # honest score including selectionThe mean and standard deviation of the outer scores are reported; a distribution instead of a single number.
Practice Nested comparison
On the breast cancer data build three inner searchers: scaled logistic ({"logisticregression__C": [0.1, 1, 10]}), random forest ({"max_depth": [None, 6]}), histogram boosting ({"learning_rate": [0.05, 0.2]}), all with cv=3. Evaluate each on the outside with cross_val_score(..., cv=5); on each line print the name and the mean with round(..., 3) (in order lr, rf, hgb).
Starter code
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import GridSearchCV, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier, HistGradientBoostingClassifier
X, y = load_breast_cancer(return_X_y=True)
# buraya yazSample solution
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import GridSearchCV, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier, HistGradientBoostingClassifier
X, y = load_breast_cancer(return_X_y=True)
aday = {
"lr": GridSearchCV(make_pipeline(StandardScaler(), LogisticRegression()), {"logisticregression__C": [0.1, 1, 10]}, cv=3),
"rf": GridSearchCV(RandomForestClassifier(random_state=0), {"max_depth": [None, 6]}, cv=3),
"hgb": GridSearchCV(HistGradientBoostingClassifier(random_state=0), {"learning_rate": [0.05, 0.2]}, cv=3),
}
for ad, ic in aday.items():
print(ad, round(cross_val_score(ic, X, y, cv=5).mean(), 3))Expected output
lr 0.975 rf 0.963 hgb 0.968
Practice Train the winner
Train the winning logistic searcher from nested CV on all the data; print the chosen C value. Then print, with round(..., 3), the accuracy on a random split's test set (test_size=0.25, random_state=0, stratify=y) of the same searcher trained on the training part.
Starter code
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
X, y = load_breast_cancer(return_X_y=True)
ic = GridSearchCV(make_pipeline(StandardScaler(), LogisticRegression()), {"logisticregression__C": [0.1, 1, 10]}, cv=3)
# buraya yazSample solution
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
X, y = load_breast_cancer(return_X_y=True)
ic = GridSearchCV(make_pipeline(StandardScaler(), LogisticRegression()), {"logisticregression__C": [0.1, 1, 10]}, cv=3)
print(ic.fit(X, y).best_params_["logisticregression__C"])
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
print(round(ic.fit(X_train, y_train).score(X_test, y_test), 3))Expected output
1 0.958
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 What is the inner loop's job in nested CV?
- Splitting the test set
- Choosing hyperparameters Correct answer
- Measuring the final score
- Scaling the data
Explanation: The inner loop selects, the outer loop measures how well the selection generalises.
2 Evaluate the inner searcher on the outside.
dis = ____(ic, X, y, cv=5)dis = cross_val_score(ic, X, y, cv=5)Explanation: GridSearchCV is an estimator too, so it can be given to cross_val_score.
3 Match the pieces.
| Term | Meaning |
|---|---|
inner CV |
Setting selection |
outer CV |
Honest measurement |
dis.std() |
Uncertainty |
inner cv=3 |
Cheaper search |