Classification

A decision tree assigns the majority class at leaf nodes.

Classifying with a tree

A new example starts at the root and reaches a leaf by going left or right according to the questions. The prediction is the majority class of the training examples in that leaf; predict_proba gives the class shares in the leaf.

m = DecisionTreeClassifier(max_depth=2, random_state=0).fit(X, y)
print(m.predict(X[:3]))
print(m.predict_proba(X[100:101]))

In depth-limited trees leaves can be mixed; the probabilities reflect this.

Feature importances

feature_importances_ shows how much each feature reduces impurity in the splits; the total is 1:

from sklearn.datasets import load_iris
d = load_iris()
m = DecisionTreeClassifier(random_state=0).fit(d.data, d.target)
for ad, o in zip(d.feature_names, m.feature_importances_):
    print(f"{ad}: {o:.2f}")

In Iris the petal measurements carry almost all of the importance; the sepal measurements contribute little.

Exercise The most important feature

Train a DecisionTreeClassifier(random_state=0) on Iris. Find the index of the most important feature with argmax and print it, then print the sum of the importances with round(..., 3).

Show example solution

from sklearn.datasets import load_iris
from sklearn.tree import DecisionTreeClassifier

X, y = load_iris(return_X_y=True)

m = DecisionTreeClassifier(random_state=0).fit(X, y)
print(m.feature_importances_.argmax())
print(round(m.feature_importances_.sum(), 3))

Test yourself

If a leaf has 8 A and 2 B examples, what is <code>predict_proba</code> for an example landing there?
- [1.0, 0.0]
- [0.5, 0.5]
- [8, 2]
- [0.8, 0.2]
Show answer

[0.8, 0.2] — The probability is the class shares in the leaf: 8/10 and 2/10.

Read the feature importances.

onem = m.____

Show answer

onem = m.feature_importances_

Match the outputs.

predict

The leaf's majority class
predict_proba

Shares in the leaf
feature_importances_

Feature contributions
get_n_leaves()

Number of leaves

Next lesson: Overfitting

We Are Enhancing Your Experience

Cookie Preferences

Strictly Necessary Cookies

Analytical Cookies