Machine Learning
Challenge 2
Evaluate a model with cross-validation.
Challenge 2: honest evaluation
The honest score of a model selection process is measured with nested CV. In this challenge you'll compare three model families (logistic, random forest, gradient boosting) in an outer CV, each tuned in an inner CV with its own small grid:
ic = GridSearchCV(RandomForestClassifier(random_state=0),
{"max_depth": [None, 6]}, cv=3)
dis = cross_val_score(ic, X, y, cv=5) # honest score including selection
The mean and standard deviation of the outer scores are reported; a distribution instead of a single number.
Exercise Nested comparison
On the breast cancer data build three inner searchers: scaled logistic ({"logisticregression__C": [0.1, 1, 10]}), random forest ({"max_depth": [None, 6]}), histogram boosting ({"learning_rate": [0.05, 0.2]}), all with cv=3. Evaluate each on the outside with cross_val_score(..., cv=5); on each line print the name and the mean with round(..., 3) (in order lr, rf, hgb).
Show example solution
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import GridSearchCV, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier, HistGradientBoostingClassifier
X, y = load_breast_cancer(return_X_y=True)
aday = {
"lr": GridSearchCV(make_pipeline(StandardScaler(), LogisticRegression()), {"logisticregression__C": [0.1, 1, 10]}, cv=3),
"rf": GridSearchCV(RandomForestClassifier(random_state=0), {"max_depth": [None, 6]}, cv=3),
"hgb": GridSearchCV(HistGradientBoostingClassifier(random_state=0), {"learning_rate": [0.05, 0.2]}, cv=3),
}
for ad, ic in aday.items():
print(ad, round(cross_val_score(ic, X, y, cv=5).mean(), 3))
Exercise Train the winner
Train the winning logistic searcher from nested CV on all the data; print the chosen C value. Then print, with round(..., 3), the accuracy on a random split's test set (test_size=0.25, random_state=0, stratify=y) of the same searcher trained on the training part.
Show example solution
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
X, y = load_breast_cancer(return_X_y=True)
ic = GridSearchCV(make_pipeline(StandardScaler(), LogisticRegression()), {"logisticregression__C": [0.1, 1, 10]}, cv=3)
print(ic.fit(X, y).best_params_["logisticregression__C"])
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
print(round(ic.fit(X_train, y_train).score(X_test, y_test), 3))
Test yourself
-
What is the inner loop's job in nested CV?
- Splitting the test set
- Choosing hyperparameters
- Measuring the final score
- Scaling the data
Show answer
Choosing hyperparameters — The inner loop selects, the outer loop measures how well the selection generalises.
-
Evaluate the inner searcher on the outside.
dis = ____(ic, X, y, cv=5)
Show answer
dis = cross_val_score(ic, X, y, cv=5)
-
Match the pieces.
- inner CV
- Setting selection
- outer CV
- Honest measurement
- dis.std()
- Uncertainty
- inner cv=3
- Cheaper search
Next lesson: Challenge 3
