Machine Learning
SMOTE
SMOTE generates synthetic samples for the minority class.
SMOTE
SMOTE (Synthetic Minority Over-sampling Technique) generates new synthetic examples between minority examples: a point is taken on the line between a minority example and a randomly chosen minority neighbour.
# imbalanced-learn library
from imblearn.over_sampling import SMOTE
X_yeni, y_yeni = SMOTE(random_state=0).fit_resample(X_train, y_train)
# the logic by hand:
yeni = a + rng.random() * (komsu - a)
The difference from plain copying is that the model sees the minority region more broadly instead of memorising the same points.
Only on training data
Resampling is applied only to the training data. If the test data had synthetic examples, the measured success wouldn't reflect reality; in CV it must also be done inside each fold. imbalanced-learn offers its own Pipeline for this:
from imblearn.pipeline import make_pipeline as imb_pipeline
pipe = imb_pipeline(StandardScaler(), SMOTE(random_state=0), LogisticRegression())
cross_val_score(pipe, X, y, cv=5, scoring="f1")
scikit-learn's own Pipeline doesn't support fit_resample steps.
Exercise SMOTE by hand
From the minority examples in the training set (X_train[y_train == 1]), generate enough synthetic examples to reach the majority count: for each new example pick a random minority example and one of its 5 nearest minority neighbours. Print the class distribution of the new training set with np.bincount.
Show example solution
import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
X, y = make_classification(n_samples=2000, n_features=10, n_informative=4, weights=[0.95, 0.05], flip_y=0, random_state=1)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)
from sklearn.neighbors import NearestNeighbors
rng = np.random.default_rng(0)
az = X_train[y_train == 1]
gerek = (y_train == 0).sum() - len(az)
_, komsu = NearestNeighbors(n_neighbors=6).fit(az).kneighbors(az)
yeni = []
for _ in range(gerek):
i = rng.integers(len(az))
j = komsu[i, rng.integers(1, 6)]
yeni.append(az[i] + rng.random() * (az[j] - az[i]))
X_yeni = np.vstack([X_train, yeni])
y_yeni = np.concatenate([y_train, np.ones(gerek, dtype=int)])
print(np.bincount(y_yeni))
Test yourself
-
Where does SMOTE place a new example?
- On the line between two minority examples
- At a random place
- At the centre of the majority class
- On top of an existing example
Show answer
On the line between two minority examples — Intermediate points fill the inside of the minority region.
-
Generate a point between a minority example and its neighbour.
yeni = a + rng.random() * (____)
Show answer
yeni = a + rng.random() * (komsu - a)
-
Match the concepts.
- SMOTE
- Synthetic minority examples
- fit_resample
- Resampling
- imblearn Pipeline
- Sampling inside CV
- test data
- Never resampled
Next lesson: Class weight
