Missing data

Missing data is detected with df.isna() and filled with df.fillna().

Finding missing values

A missing value appears as NaN in pandas. First you check where and how many there are:

print(df.isna().sum())        # missing count per column
print(df.isna().mean() * 100) # as a percentage

Most scikit-learn models can't be trained on data containing NaN and raise a ValueError, so missing values must be handled.

Drop or fill?

If few rows have missing values they can be dropped; if many, the data loss grows and filling is better:

df.dropna()                          # drop rows with missing values
df["yas"].fillna(df["yas"].median()) # fill with the median

from sklearn.impute import SimpleImputer
imp = SimpleImputer(strategy="mean")
X = imp.fit_transform(X)

SimpleImputer applies the value learned from the training data to the test data as well, so test information doesn't leak into training.

Exercise Fill the missing values

Fill the np.nan values in X with SimpleImputer(strategy="mean") and print the result.

Show example solution

import numpy as np
from sklearn.impute import SimpleImputer

X = np.array([[1, 10], [np.nan, 20], [3, np.nan]])

imp = SimpleImputer(strategy="mean")
print(imp.fit_transform(X))

Test yourself

What happens if <code>LinearRegression().fit</code> is called with data containing missing values?
- It treats missing values as zero
- It skips the missing rows
- It warns and continues
- It raises a ValueError
Show answer

It raises a ValueError — Missing values must be dropped or filled first.
Create the object that fills missing values with the column mean.
```
imp = SimpleImputer(strategy=____)
```
Show answer
```
imp = SimpleImputer(strategy="mean")
```
Match the tools.

isna()

Marks missing values
dropna()

Drops rows with missing values
fillna()

Fills missing values
SimpleImputer

Trainable filler

Next lesson: Data types

We Are Enhancing Your Experience

Cookie Preferences

Strictly Necessary Cookies

Analytical Cookies