Data cleaning

Missing data (NaN) is detected and filled or removed.

Why is data cleaned?

Real data is messy: duplicate rows, spelling differences ("ankara", "Ankara "), meaningless values (age = -3). The model can't tell them apart: "garbage in, garbage out".

df = df.drop_duplicates()                       # drop duplicates
df["sehir"] = df["sehir"].str.strip().str.title() # spaces and case
df = df[df["yas"].between(0, 120)]              # drop meaningless ages

Cleaning takes most of the time in most projects and makes more difference than the choice of model.

Outliers

Values very far from the others are called outliers. A common rule is based on the interquartile range (IQR):

x = np.array([10, 12, 11, 13, 12, 95])
q1, q3 = np.percentile(x, [25, 75])
iqr = q3 - q1
temiz = x[(x >= q1 - 1.5 * iqr) & (x <= q3 + 1.5 * iqr)]
print(temiz)   # [10 12 11 13 12]

An outlier isn't always an error (it might really be a very expensive house); understand the reason before dropping it.

Exercise Remove the outlier

Find the 25th and 75th percentiles of x with np.percentile, drop the outliers with the IQR rule (1.5 times) and print the remaining array.

Show example solution

import numpy as np

x = np.array([5, 7, 6, 8, 7, -40, 6])

q1, q3 = np.percentile(x, [25, 75])
iqr = q3 - q1
print(x[(x >= q1 - 1.5 * iqr) & (x <= q3 + 1.5 * iqr)])

Test yourself

What does the "garbage in, garbage out" principle say?
- A model trained on bad data gives bad results
- The model fixes bad data itself
- The more data the better, always
- Cleaning is unnecessary
Show answer

A model trained on bad data gives bad results — The model learns only from the data it is given; faulty data means faulty patterns.
Drop the duplicate rows.
```
df = df.____()
```
Show answer
```
df = df.drop_duplicates()
```
Match the problems with their fixes.

Duplicate row

drop_duplicates
Extra whitespace

str.strip
Letter case difference

str.lower
Outlier

IQR rule

Next lesson: Missing data

We Are Enhancing Your Experience

Cookie Preferences

Strictly Necessary Cookies

Analytical Cookies