Cleaning data

Detect and clean missing data.

Filling the gaps

There are two empty cells: the humidity of day 2 and the temperature of day 3. Dropping the rows would lose 40% of the data; instead we'll fill each column with its own mean. np.nanmean takes the mean while ignoring nans:

print(np.nanmean(veri, axis=0))   # [ 3.   23.   48.75]
print(veri.mean(axis=0))          # [ 3.  nan  nan]  nan is contagious

Any operation involving nan gives nan, which is why nan-safe functions are needed.

Filling selectively with np.where

np.where(condition, a, b) takes from a where the condition is true and from b otherwise. Thanks to broadcasting, the column means are applied to every row:

ort = np.nanmean(veri, axis=0)
veri = np.where(np.isnan(veri), ort, veri)
print(np.isnan(veri).sum())   # 0

In scikit-learn, SimpleImputer(strategy="mean") does the same; here you saw its logic by hand.

Exercise Clean

Fill the nan values in veri using np.nanmean(veri, axis=0) and np.where. Print the temperature column (veri[:, 1]) and the humidity column (veri[:, 2]) on separate lines.

Show example solution

import io
import numpy as np

csv = "gun,sicaklik,nem\n1,20,40\n2,22,\n3,,60\n4,26,50\n5,24,45\n"
veri = np.genfromtxt(io.StringIO(csv), delimiter=",", skip_header=1)

ort = np.nanmean(veri, axis=0)
veri = np.where(np.isnan(veri), ort, veri)
print(veri[:, 1])
print(veri[:, 2])

Test yourself

What does <code>np.array([1, np.nan, 3]).mean()</code> give?
- 2.0
- 1.33
- An error
- nan
Show answer

nan — A sum containing nan becomes nan; nanmean would give 2.0.

Take the column means ignoring nan values.

ort = np.____(veri, axis=0)

Show answer

ort = np.nanmean(veri, axis=0)

Match the functions.

np.nanmean

Mean without nan
np.where

Conditional selection
np.isnan

nan mask
x.mean()

Returns nan if nan exists

Next lesson: Computing statistics

We Are Enhancing Your Experience

Cookie Preferences

Strictly Necessary Cookies

Analytical Cookies