CodeBlanks Logo Icon CodeBlanks
Login Register

Splitting criterion

Decision trees split data using the gini or entropy criterion.

Gini impurity

At each step the tree chooses the question that makes the nodes most "pure". The default purity measure is Gini: 1 - Σ pᵢ². In a node of a single class Gini is 0; if two classes are mixed evenly it is 0.5.

def gini(p):
    p = np.array(p)
    return 1 - (p ** 2).sum()
print(gini([1.0, 0.0]))   # 0.0  pure
print(gini([0.5, 0.5]))   # 0.5  most mixed

The value of a split is how much the weighted mean Gini of the child nodes drops below the parent's.

Entropy

The other option is entropy, from information theory: -Σ pᵢ log₂ pᵢ. It is 0 in a pure node and 1 if two classes are equal. A split's drop in entropy is called information gain.

m = DecisionTreeClassifier(criterion="entropy", random_state=0)

In practice Gini and entropy often produce the same tree; Gini is slightly faster because it has no logarithm.

Practice Compute Gini

Write a gini(etiketler) function giving the Gini value of a list of labels (find the class shares with np.unique(..., return_counts=True)). Print the results for [0, 0, 1, 1] and [0, 0, 0, 1].

Starter code

model.py
import numpy as np

# buraya yaz

Sample solution

model.py
import numpy as np

def gini(etiketler):
    _, sayi = np.unique(etiketler, return_counts=True)
    p = sayi / sayi.sum()
    return 1 - (p ** 2).sum()

print(gini([0, 0, 1, 1]))
print(gini([0, 0, 0, 1]))

Expected output

0.5
0.375

Questions and answers

Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.

1 What is the Gini value of a node with class shares 0.75 and 0.25?

  • 0.625
  • 0.375 Correct answer
  • 0.5
  • 0.25

Explanation: 1 - (0.75² + 0.25²) = 1 - (0.5625 + 0.0625) = 0.375.

2 Choose entropy as the split criterion.

Question 
m = DecisionTreeClassifier(____="entropy")
Answer 
m = DecisionTreeClassifier(criterion="entropy")

Explanation: criterion can be "gini" (default), "entropy" or "log_loss".

3 Match the values.

Term Meaning
Gini = 0 Pure node
Gini = 0.5 Two classes evenly mixed
Entropi = 1 Two classes equal (log₂)
information gain Drop in entropy

All lessons in Beginner

1 What is ML?

  1. What is machine learning?
  2. Supervised and unsupervised
  3. Data and model
  4. Training and testing
  5. Feature and label notation
  6. Summary - ML fundamentals

2 NumPy basics

  1. What is NumPy?
  2. Creating arrays
  3. Array properties
  4. Array indexing
  5. Reshaping arrays
  6. Summary - NumPy basics

3 Pandas basics

  1. What is Pandas?
  2. Creating a DataFrame
  3. Column operations
  4. Selecting and filtering data
  5. Viewing first rows
  6. Summary - Pandas basics

4 Data preprocessing

  1. Data cleaning
  2. Missing data
  3. Data types
  4. Label encoding
  5. Data splitting
  6. Summary - Data preprocessing

5 Statistics basics

  1. Mean
  2. Median
  3. Standard deviation
  4. Variance
  5. Correlation
  6. Summary - Statistics

6 Linear algebra

  1. Vectors
  2. Matrices
  3. Matrix multiplication
  4. Transpose
  5. Identity matrix
  6. Summary - Linear algebra

7 Project: Data analysis

  1. Loading data
  2. Cleaning data
  3. Computing statistics
  4. Printing results

8 Regression

  1. What is regression?
  2. Linear regression
  3. Model training
  4. Prediction
  5. Error measurement
  6. Summary - Regression

9 KNN

  1. What is KNN?
  2. The K parameter
  3. Distance measurement
  4. Classification
  5. Model training
  6. Summary - KNN

10 Decision trees

  1. What is a decision tree?
  2. Splitting criterion
  3. Depth
  4. Classification
  5. Overfitting
  6. Summary - Decision trees

11 Classification

  1. What is classification?
  2. Binary classification
  3. Multi-class
  4. Accuracy
  5. Confusion matrix
  6. Summary - Classification

12 Model evaluation

  1. Train-test split
  2. Accuracy
  3. Precision
  4. Recall
  5. F1 score
  6. Summary - Model evaluation

13 Project: Classification

  1. Preparing data
  2. Training model
  3. Prediction
  4. Evaluation

14 Scaling

  1. What is scaling?
  2. StandardScaler
  3. MinMaxScaler
  4. When to use?
  5. Pipeline
  6. Summary - Scaling

15 Feature engineering

  1. What is a feature?
  2. Feature selection
  3. Feature extraction
  4. Categorical features
  5. Numerical features
  6. Summary - Feature engineering

16 Model selection

  1. What is model selection?
  2. Cross-validation
  3. Grid search
  4. Score comparison
  5. Hyperparameter
  6. Summary - Model selection

17 Project: Regression

  1. Preparing data
  2. Training model
  3. Prediction
  4. Error measurement

18 Metrics

  1. What is a metric?
  2. Accuracy
  3. Precision
  4. Recall
  5. F1 score
  6. Summary - Metrics

19 Data visualization

  1. What is visualization?
  2. Scatter plot
  3. Histogram
  4. Line plot
  5. Box plot
  6. Summary - Visualization

20 Beginner final

  1. Challenge 1
  2. Challenge 2
  3. Challenge 3
  4. Challenge 4