SMOTE
- Machine Learning course
- Intermediate
- Lesson 88/112
- 5 min
SMOTE generates synthetic samples for the minority class.
SMOTE
SMOTE (Synthetic Minority Over-sampling Technique) generates new synthetic examples between minority examples: a point is taken on the line between a minority example and a randomly chosen minority neighbour.
# imbalanced-learn library
from imblearn.over_sampling import SMOTE
X_yeni, y_yeni = SMOTE(random_state=0).fit_resample(X_train, y_train)
# the logic by hand:
yeni = a + rng.random() * (komsu - a)The difference from plain copying is that the model sees the minority region more broadly instead of memorising the same points.
Only on training data
Resampling is applied only to the training data. If the test data had synthetic examples, the measured success wouldn't reflect reality; in CV it must also be done inside each fold. imbalanced-learn offers its own Pipeline for this:
from imblearn.pipeline import make_pipeline as imb_pipeline
pipe = imb_pipeline(StandardScaler(), SMOTE(random_state=0), LogisticRegression())
cross_val_score(pipe, X, y, cv=5, scoring="f1")scikit-learn's own Pipeline doesn't support fit_resample steps.
Practice SMOTE by hand
From the minority examples in the training set (X_train[y_train == 1]), generate enough synthetic examples to reach the majority count: for each new example pick a random minority example and one of its 5 nearest minority neighbours. Print the class distribution of the new training set with np.bincount.
Starter code
import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
X, y = make_classification(n_samples=2000, n_features=10, n_informative=4, weights=[0.95, 0.05], flip_y=0, random_state=1)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)
from sklearn.neighbors import NearestNeighbors
rng = np.random.default_rng(0)
# buraya yazSample solution
import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
X, y = make_classification(n_samples=2000, n_features=10, n_informative=4, weights=[0.95, 0.05], flip_y=0, random_state=1)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)
from sklearn.neighbors import NearestNeighbors
rng = np.random.default_rng(0)
az = X_train[y_train == 1]
gerek = (y_train == 0).sum() - len(az)
_, komsu = NearestNeighbors(n_neighbors=6).fit(az).kneighbors(az)
yeni = []
for _ in range(gerek):
i = rng.integers(len(az))
j = komsu[i, rng.integers(1, 6)]
yeni.append(az[i] + rng.random() * (az[j] - az[i]))
X_yeni = np.vstack([X_train, yeni])
y_yeni = np.concatenate([y_train, np.ones(gerek, dtype=int)])
print(np.bincount(y_yeni))Expected output
[1330 1330]
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 Where does SMOTE place a new example?
- On the line between two minority examples Correct answer
- At a random place
- At the centre of the majority class
- On top of an existing example
Explanation: Intermediate points fill the inside of the minority region.
2 Generate a point between a minority example and its neighbour.
yeni = a + rng.random() * (____)yeni = a + rng.random() * (komsu - a)Explanation: Move from a towards the neighbour by a random ratio.
3 Match the concepts.
| Term | Meaning |
|---|---|
SMOTE |
Synthetic minority examples |
fit_resample |
Resampling |
imblearn Pipeline |
Sampling inside CV |
test data |
Never resampled |