Challenge 4
- Machine Learning course
- Intermediate
- Lesson 112/112
- 8 min
Reduce dimensions with PCA.
Challenge 4: imbalanced end to end
The final challenge is the hardest scenario of the intermediate level: maximising F1 on data with a 3% minority and missing values. The tools you'll use:
HistGradientBoostingClassifier(NaN-capable) andclass_weight.- Searching by F1 with
StratifiedKFold. - Threshold tuning on a validation set.
- A single final measurement on the test set.
Practice Model and weight
On the given data, compare HistGradientBoostingClassifier(random_state=0) with class_weight None and "balanced" by the mean 5-fold CV F1 on the training part. Print the two scores with round(..., 3) on one line.
Starter code
import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.ensemble import HistGradientBoostingClassifier
X, y = make_classification(n_samples=3000, n_features=15, n_informative=6, weights=[0.97, 0.03], random_state=3)
rng = np.random.default_rng(3)
X[rng.random(X.shape) < 0.05] = np.nan
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
# buraya yazSample solution
import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.ensemble import HistGradientBoostingClassifier
X, y = make_classification(n_samples=3000, n_features=15, n_informative=6, weights=[0.97, 0.03], random_state=3)
rng = np.random.default_rng(3)
X[rng.random(X.shape) < 0.05] = np.nan
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
sonuc = []
for cw in [None, "balanced"]:
m = HistGradientBoostingClassifier(class_weight=cw, random_state=0)
sonuc.append(round(cross_val_score(m, X_train, y_train, cv=5, scoring="f1").mean(), 3))
print(*sonuc)Expected output
0.365 0.481
Practice Threshold and final measurement
Split the training part again with test_size=0.3, random_state=1, stratify=y_train. Train the class_weight="balanced" model on the training part and find the threshold maximising F1 on validation. Print the test F1 at the 0.5 threshold and at the found threshold with round(..., 3) on one line.
Starter code
import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.metrics import f1_score, precision_recall_curve
X, y = make_classification(n_samples=3000, n_features=15, n_informative=6, weights=[0.97, 0.03], random_state=3)
rng = np.random.default_rng(3)
X[rng.random(X.shape) < 0.05] = np.nan
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
# buraya yazSample solution
import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.metrics import f1_score, precision_recall_curve
X, y = make_classification(n_samples=3000, n_features=15, n_informative=6, weights=[0.97, 0.03], random_state=3)
rng = np.random.default_rng(3)
X[rng.random(X.shape) < 0.05] = np.nan
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
X_tr, X_val, y_tr, y_val = train_test_split(X_train, y_train, test_size=0.3, random_state=1, stratify=y_train)
m = HistGradientBoostingClassifier(class_weight="balanced", random_state=0).fit(X_tr, y_tr)
pr, re, esik = precision_recall_curve(y_val, m.predict_proba(X_val)[:, 1])
f1 = 2 * pr * re / (pr + re + 1e-12)
en_iyi = esik[f1[:-1].argmax()]
p = m.predict_proba(X_test)[:, 1]
print(round(f1_score(y_test, p >= 0.5), 3), round(f1_score(y_test, p >= en_iyi), 3))Expected output
0.537 0.638
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 Why is the threshold tuned on a validation set rather than the test set?
- The test set is too small
- It isn't a rule
- It is faster on validation
- A threshold tuned on the test makes the test score optimistic Correct answer
Explanation: The threshold is a hyperparameter too; test data must not take part in its choice.
2 Balance the class weights.
Question
m = HistGradientBoostingClassifier(class_weight=____, random_state=0)
Answer
m = HistGradientBoostingClassifier(class_weight="balanced", random_state=0)Explanation: HistGradientBoosting also supports the class_weight parameter.
3 Match the tools with the jobs.
| Term | Meaning |
|---|---|
HistGradientBoosting |
Model with NaN data |
class_weight |
Weight on the minority |
threshold tuning |
Raises F1 |
test set |
A single final measurement |