Summary - Data preprocessing

In this group you learned data cleaning, missing data, and label encoding.

Summary: data preprocessing

The steps data goes through before the model:

Cleaning: duplicates, spelling differences, outliers.
Missing values: dropna or SimpleImputer.
Types: to_numeric, astype.
Encoding: LabelEncoder for labels, one-hot for unordered features.
Splitting: train_test_split, with stratify and a validation set when needed.

The golden rule: every learned step such as filling and encoding is fit only on the training data.

Exercise Leak-free filling

fit a SimpleImputer(strategy="mean") on X_train only, then transform X_test and print the result.

Show example solution

import numpy as np
from sklearn.impute import SimpleImputer

X_train = np.array([[2.0], [4.0], [6.0]])
X_test = np.array([[np.nan], [10.0]])

imp = SimpleImputer(strategy="mean")
imp.fit(X_train)
print(imp.transform(X_test))

Test yourself

How should SimpleImputer be used on the test data?
- fit_transform again on the test data
- It shouldn't be used at all
- It should be fitted on all the data
- Only transform, with the object fitted on training
Show answer

Only transform, with the object fitted on training — The test data's mean must not leak into training; the same learned value is applied.
Apply the filling learned on training to the test data.
```
X_test = imp.____(X_test)
```
Show answer
```
X_test = imp.transform(X_test)
```
Match the steps with the tools.

Filling missing values

SimpleImputer
Text label

LabelEncoder
Unordered feature

OneHotEncoder
Balanced split

stratify

Next lesson: Mean

We Are Enhancing Your Experience

Cookie Preferences

Strictly Necessary Cookies

Analytical Cookies