Missing data
- Machine Learning course
- Beginner
- Lesson 20/112
- 5 min
Missing data is detected with df.isna() and filled with df.fillna().
Finding missing values
A missing value appears as NaN in pandas. First you check where and how many there are:
print(df.isna().sum()) # missing count per column
print(df.isna().mean() * 100) # as a percentageMost scikit-learn models can't be trained on data containing NaN and raise a ValueError, so missing values must be handled.
Drop or fill?
If few rows have missing values they can be dropped; if many, the data loss grows and filling is better:
df.dropna() # drop rows with missing values
df["yas"].fillna(df["yas"].median()) # fill with the median
from sklearn.impute import SimpleImputer
imp = SimpleImputer(strategy="mean")
X = imp.fit_transform(X)SimpleImputer applies the value learned from the training data to the test data as well, so test information doesn't leak into training.
Practice Fill the missing values
Fill the np.nan values in X with SimpleImputer(strategy="mean") and print the result.
Starter code
import numpy as np
from sklearn.impute import SimpleImputer
X = np.array([[1, 10], [np.nan, 20], [3, np.nan]])
# buraya yazSample solution
import numpy as np
from sklearn.impute import SimpleImputer
X = np.array([[1, 10], [np.nan, 20], [3, np.nan]])
imp = SimpleImputer(strategy="mean")
print(imp.fit_transform(X))Expected output
[[ 1. 10.] [ 2. 20.] [ 3. 15.]]
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 What happens if <code>LinearRegression().fit</code> is called with data containing missing values?
- It treats missing values as zero
- It skips the missing rows
- It warns and continues
- It raises a ValueError Correct answer
Explanation: Missing values must be dropped or filled first.
2 Create the object that fills missing values with the column mean.
imp = SimpleImputer(strategy=____)imp = SimpleImputer(strategy="mean")Explanation: strategy can be "mean", "median", "most_frequent" or "constant".
3 Match the tools.
| Term | Meaning |
|---|---|
isna() |
Marks missing values |
dropna() |
Drops rows with missing values |
fillna() |
Fills missing values |
SimpleImputer |
Trainable filler |