CodeBlanks Logo Icon CodeBlanks
Login Register

Data cleaning

Missing data (NaN) is detected and filled or removed.

Why is data cleaned?

Real data is messy: duplicate rows, spelling differences ("ankara", "Ankara "), meaningless values (age = -3). The model can't tell them apart: "garbage in, garbage out".

df = df.drop_duplicates()                       # drop duplicates
df["sehir"] = df["sehir"].str.strip().str.title() # spaces and case
df = df[df["yas"].between(0, 120)]              # drop meaningless ages

Cleaning takes most of the time in most projects and makes more difference than the choice of model.

Outliers

Values very far from the others are called outliers. A common rule is based on the interquartile range (IQR):

x = np.array([10, 12, 11, 13, 12, 95])
q1, q3 = np.percentile(x, [25, 75])
iqr = q3 - q1
temiz = x[(x >= q1 - 1.5 * iqr) & (x <= q3 + 1.5 * iqr)]
print(temiz)   # [10 12 11 13 12]

An outlier isn't always an error (it might really be a very expensive house); understand the reason before dropping it.

Practice Remove the outlier

Find the 25th and 75th percentiles of x with np.percentile, drop the outliers with the IQR rule (1.5 times) and print the remaining array.

Starter code

model.py
import numpy as np

x = np.array([5, 7, 6, 8, 7, -40, 6])

# buraya yaz

Sample solution

model.py
import numpy as np

x = np.array([5, 7, 6, 8, 7, -40, 6])

q1, q3 = np.percentile(x, [25, 75])
iqr = q3 - q1
print(x[(x >= q1 - 1.5 * iqr) & (x <= q3 + 1.5 * iqr)])

Expected output

[5 7 6 8 7 6]

Questions and answers

Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.

1 What does the "garbage in, garbage out" principle say?

  • A model trained on bad data gives bad results Correct answer
  • The model fixes bad data itself
  • The more data the better, always
  • Cleaning is unnecessary

Explanation: The model learns only from the data it is given; faulty data means faulty patterns.

2 Drop the duplicate rows.

Question 
df = df.____()
Answer 
df = df.drop_duplicates()

Explanation: drop_duplicates keeps only the first of identical rows.

3 Match the problems with their fixes.

Term Meaning
Duplicate row drop_duplicates
Extra whitespace str.strip
Letter case difference str.lower
Outlier IQR rule

All lessons in Beginner

1 What is ML?

  1. What is machine learning?
  2. Supervised and unsupervised
  3. Data and model
  4. Training and testing
  5. Feature and label notation
  6. Summary - ML fundamentals

2 NumPy basics

  1. What is NumPy?
  2. Creating arrays
  3. Array properties
  4. Array indexing
  5. Reshaping arrays
  6. Summary - NumPy basics

3 Pandas basics

  1. What is Pandas?
  2. Creating a DataFrame
  3. Column operations
  4. Selecting and filtering data
  5. Viewing first rows
  6. Summary - Pandas basics

4 Data preprocessing

  1. Data cleaning
  2. Missing data
  3. Data types
  4. Label encoding
  5. Data splitting
  6. Summary - Data preprocessing

5 Statistics basics

  1. Mean
  2. Median
  3. Standard deviation
  4. Variance
  5. Correlation
  6. Summary - Statistics

6 Linear algebra

  1. Vectors
  2. Matrices
  3. Matrix multiplication
  4. Transpose
  5. Identity matrix
  6. Summary - Linear algebra

7 Project: Data analysis

  1. Loading data
  2. Cleaning data
  3. Computing statistics
  4. Printing results

8 Regression

  1. What is regression?
  2. Linear regression
  3. Model training
  4. Prediction
  5. Error measurement
  6. Summary - Regression

9 KNN

  1. What is KNN?
  2. The K parameter
  3. Distance measurement
  4. Classification
  5. Model training
  6. Summary - KNN

10 Decision trees

  1. What is a decision tree?
  2. Splitting criterion
  3. Depth
  4. Classification
  5. Overfitting
  6. Summary - Decision trees

11 Classification

  1. What is classification?
  2. Binary classification
  3. Multi-class
  4. Accuracy
  5. Confusion matrix
  6. Summary - Classification

12 Model evaluation

  1. Train-test split
  2. Accuracy
  3. Precision
  4. Recall
  5. F1 score
  6. Summary - Model evaluation

13 Project: Classification

  1. Preparing data
  2. Training model
  3. Prediction
  4. Evaluation

14 Scaling

  1. What is scaling?
  2. StandardScaler
  3. MinMaxScaler
  4. When to use?
  5. Pipeline
  6. Summary - Scaling

15 Feature engineering

  1. What is a feature?
  2. Feature selection
  3. Feature extraction
  4. Categorical features
  5. Numerical features
  6. Summary - Feature engineering

16 Model selection

  1. What is model selection?
  2. Cross-validation
  3. Grid search
  4. Score comparison
  5. Hyperparameter
  6. Summary - Model selection

17 Project: Regression

  1. Preparing data
  2. Training model
  3. Prediction
  4. Error measurement

18 Metrics

  1. What is a metric?
  2. Accuracy
  3. Precision
  4. Recall
  5. F1 score
  6. Summary - Metrics

19 Data visualization

  1. What is visualization?
  2. Scatter plot
  3. Histogram
  4. Line plot
  5. Box plot
  6. Summary - Visualization

20 Beginner final

  1. Challenge 1
  2. Challenge 2
  3. Challenge 3
  4. Challenge 4