Machine Learning
Class weight
The class_weight parameter gives more weight to the minority class.
class_weight
Instead of changing the data, the minority's errors can be given more weight in the model's loss function. class_weight="balanced" weights each class by n / (k · n_class):
m = LogisticRegression(class_weight="balanced").fit(X_train, y_train)
from sklearn.utils.class_weight import compute_class_weight
print(compute_class_weight("balanced", classes=np.array([0, 1]), y=y_train))
# [0.526 10.] a minority error is ~19 times costlier
Because it generates no new examples it is fast, and it is available directly in most scikit-learn models (logistic, SVM, trees, forests).
Manual weights
The weights can also be given manually with a dict, so the real business cost of errors is reflected in the model. For example, if missing a fraud is 50 times costlier than a false alarm:
m = LogisticRegression(class_weight={0: 1, 1: 50})
# or as example weights in fit:
w = np.where(y_train == 1, 50, 1)
m.fit(X_train, y_train, sample_weight=w)
As the weight grows recall rises and precision falls; the best value is chosen by validation.
Exercise Weighted model
Train an unweighted and a class_weight="balanced" logistic regression. For each print the test precision and recall with round(..., 3) on one line (unweighted first).
Show example solution
import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
X, y = make_classification(n_samples=2000, n_features=10, n_informative=4, weights=[0.95, 0.05], flip_y=0, random_state=1)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import precision_score, recall_score
for cw in [None, "balanced"]:
p = LogisticRegression(class_weight=cw).fit(X_train, y_train).predict(X_test)
print(round(precision_score(y_test, p), 3), round(recall_score(y_test, p), 3))
Test yourself
-
With 1000 examples of which 900 majority and 100 minority, what are the balanced weights?
- 1 and 9
- 0.9 and 0.1
- 1 and 1
- 0.556 and 5.0
Show answer
0.556 and 5.0 — 1000 / (2 · 900) ≈ 0.556; 1000 / (2 · 100) = 5.
-
Give the minority class 20 times the weight.
m = LogisticRegression(class_weight=____)
Show answer
m = LogisticRegression(class_weight={0: 1, 1: 20})
-
Match the weighting forms.
- "balanced"
- n / (k · n_class)
- {0: 1, 1: 50}
- Manual cost
- sample_weight
- Per-example weight
- compute_class_weight
- Computes the weights
Next lesson: Oversampling
