Machine Learning
Overfitting
Very deep trees memorize data; pruning prevents this.
Overfitting
If the model also learns the noise in the training data, it does very well in training and badly on new data; this is called overfitting. Unlimited trees are its best-known example:
m = DecisionTreeClassifier(random_state=0).fit(X_train, y_train)
print(m.score(X_train, y_train)) # 1.0
print(m.score(X_test, y_test)) # noticeably lower
A large gap between training and test scores is the sign of overfitting.
Pruning
The fix is to keep the tree small: max_depth, min_samples_leaf or cost-complexity pruning (ccp_alpha). As ccp_alpha grows, branches that contribute little are cut:
for a in [0.0, 0.01, 0.05]:
m = DecisionTreeClassifier(ccp_alpha=a, random_state=0).fit(X_train, y_train)
print(a, m.get_n_leaves(), m.score(X_test, y_test))
The opposite is a problem too: a too-simple model (for example depth 1) is bad both in training and in testing; this is called underfitting.
Exercise See overfitting
On the noisy data, train an unlimited DecisionTreeClassifier(random_state=0) and a tree with max_depth=2. For each, print the training and test accuracy on the same line with round(..., 2) (unlimited first).
Show example solution
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
rng = np.random.default_rng(0)
X = rng.uniform(0, 10, size=(200, 2))
y = (X[:, 0] + rng.normal(0, 2, 200) > 5).astype(int)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)
for m in [DecisionTreeClassifier(random_state=0), DecisionTreeClassifier(max_depth=2, random_state=0)]:
m.fit(X_train, y_train)
print(round(m.score(X_train, y_train), 2), round(m.score(X_test, y_test), 2))
Test yourself
-
If training accuracy is 0.99 and test accuracy 0.70, what has happened?
- No leakage, the model is good
- The test set is faulty
- Overfitting
- Underfitting
Show answer
Overfitting — The model has memorised the training data and doesn't generalise.
-
Apply cost-complexity pruning.
m = DecisionTreeClassifier(____=0.02)
Show answer
m = DecisionTreeClassifier(ccp_alpha=0.02)
-
Match the situations.
- High training, low test
- Overfitting
- Both low
- Underfitting
- ccp_alpha
- Pruning
- min_samples_leaf
- Prevents tiny leaves
Next lesson: Summary - Decision trees
