Splitting criterion

Decision trees split data using the gini or entropy criterion.

Gini impurity

At each step the tree chooses the question that makes the nodes most "pure". The default purity measure is Gini: 1 - Σ pᵢ². In a node of a single class Gini is 0; if two classes are mixed evenly it is 0.5.

def gini(p):
    p = np.array(p)
    return 1 - (p ** 2).sum()
print(gini([1.0, 0.0]))   # 0.0  pure
print(gini([0.5, 0.5]))   # 0.5  most mixed

The value of a split is how much the weighted mean Gini of the child nodes drops below the parent's.

Entropy

The other option is entropy, from information theory: -Σ pᵢ log₂ pᵢ. It is 0 in a pure node and 1 if two classes are equal. A split's drop in entropy is called information gain.

m = DecisionTreeClassifier(criterion="entropy", random_state=0)

In practice Gini and entropy often produce the same tree; Gini is slightly faster because it has no logarithm.

Exercise Compute Gini

Write a gini(etiketler) function giving the Gini value of a list of labels (find the class shares with np.unique(..., return_counts=True)). Print the results for [0, 0, 1, 1] and [0, 0, 0, 1].

Show example solution

import numpy as np

def gini(etiketler):
    _, sayi = np.unique(etiketler, return_counts=True)
    p = sayi / sayi.sum()
    return 1 - (p ** 2).sum()

print(gini([0, 0, 1, 1]))
print(gini([0, 0, 0, 1]))

Test yourself

What is the Gini value of a node with class shares 0.75 and 0.25?
- 0.625
- 0.375
- 0.5
- 0.25
Show answer

0.375 — 1 - (0.75² + 0.25²) = 1 - (0.5625 + 0.0625) = 0.375.

Choose entropy as the split criterion.

m = DecisionTreeClassifier(____="entropy")

Show answer

m = DecisionTreeClassifier(criterion="entropy")

Match the values.

Gini = 0

Pure node
Gini = 0.5

Two classes evenly mixed
Entropi = 1

Two classes equal (log₂)
information gain

Drop in entropy

Next lesson: Depth

We Are Enhancing Your Experience

Cookie Preferences

Strictly Necessary Cookies

Analytical Cookies