Oversampling
- Machine Learning course
- Intermediate
- Lesson 90/112
- 5 min
Oversampling duplicates minority class samples.
Oversampling and undersampling
There are two basic resampling directions:
- Oversampling: multiplies the minority (random copying or SMOTE). No information is lost but the risk of overfitting rises.
- Undersampling: reduces the majority. Fast, but part of the majority's information is thrown away.
az = np.where(y_train == 1)[0]
cok = np.where(y_train == 0)[0]
# oversampling: draw from the minority with replacement
ek = rng.choice(az, size=len(cok) - len(az), replace=True)
# undersampling: draw as many from the majority as the minority has
sec = rng.choice(cok, size=len(az), replace=False)Comparison inside CV
Which strategy is better depends on the data; for a fair comparison the sampling must be done in the training part of each fold and the validation part left untouched. F1 or average precision (average_precision) is used as the metric:
for egitim, dogrulama in StratifiedKFold(5).split(X, y):
Xr, yr = yeniden_ornekle(X[egitim], y[egitim])
m.fit(Xr, yr)
skor.append(f1_score(y[dogrulama], m.predict(X[dogrulama])))In most cases class weights and threshold tuning are simpler and good enough compared with sampling.
Practice Undersampling
Randomly select as many majority examples as the minority has, without replacement, and combine them with the minority. Print the class distribution of the balanced training set, then the test recall of a logistic regression trained on it with round(..., 3).
Starter code
import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
X, y = make_classification(n_samples=2000, n_features=10, n_informative=4, weights=[0.95, 0.05], flip_y=0, random_state=1)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import recall_score
rng = np.random.default_rng(0)
# buraya yazSample solution
import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
X, y = make_classification(n_samples=2000, n_features=10, n_informative=4, weights=[0.95, 0.05], flip_y=0, random_state=1)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import recall_score
rng = np.random.default_rng(0)
az = np.where(y_train == 1)[0]
cok = np.where(y_train == 0)[0]
sec = np.concatenate([az, rng.choice(cok, size=len(az), replace=False)])
print(np.bincount(y_train[sec]))
m = LogisticRegression().fit(X_train[sec], y_train[sec])
print(round(recall_score(y_test, m.predict(X_test)), 3))Expected output
[70 70] 0.9
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 What is the main risk of random oversampling?
- Less data
- There is no risk
- Memorising the same minority examples Correct answer
- Information loss
Explanation: Copied points present the same information to the model again and again.
2 Draw examples from the minority with replacement.
ek = rng.choice(az, size=n, replace=____)ek = rng.choice(az, size=n, replace=True)Explanation: replace=True allows the same index to be picked more than once.
3 Match the strategies.
| Term | Meaning |
|---|---|
oversampling |
Multiply the minority |
undersampling |
Reduce the majority |
SMOTE |
Synthetic intermediate examples |
class_weight |
Without changing data |