Overfitting
- Machine Learning course
- Beginner
- Lesson 57/112
- 5 min
Very deep trees memorize data; pruning prevents this.
Overfitting
If the model also learns the noise in the training data, it does very well in training and badly on new data; this is called overfitting. Unlimited trees are its best-known example:
m = DecisionTreeClassifier(random_state=0).fit(X_train, y_train)
print(m.score(X_train, y_train)) # 1.0
print(m.score(X_test, y_test)) # noticeably lowerA large gap between training and test scores is the sign of overfitting.
Pruning
The fix is to keep the tree small: max_depth, min_samples_leaf or cost-complexity pruning (ccp_alpha). As ccp_alpha grows, branches that contribute little are cut:
for a in [0.0, 0.01, 0.05]:
m = DecisionTreeClassifier(ccp_alpha=a, random_state=0).fit(X_train, y_train)
print(a, m.get_n_leaves(), m.score(X_test, y_test))The opposite is a problem too: a too-simple model (for example depth 1) is bad both in training and in testing; this is called underfitting.
Practice See overfitting
On the noisy data, train an unlimited DecisionTreeClassifier(random_state=0) and a tree with max_depth=2. For each, print the training and test accuracy on the same line with round(..., 2) (unlimited first).
Starter code
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
rng = np.random.default_rng(0)
X = rng.uniform(0, 10, size=(200, 2))
y = (X[:, 0] + rng.normal(0, 2, 200) > 5).astype(int)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)
# buraya yazSample solution
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
rng = np.random.default_rng(0)
X = rng.uniform(0, 10, size=(200, 2))
y = (X[:, 0] + rng.normal(0, 2, 200) > 5).astype(int)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)
for m in [DecisionTreeClassifier(random_state=0), DecisionTreeClassifier(max_depth=2, random_state=0)]:
m.fit(X_train, y_train)
print(round(m.score(X_train, y_train), 2), round(m.score(X_test, y_test), 2))Expected output
1.0 0.78 0.85 0.82
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 If training accuracy is 0.99 and test accuracy 0.70, what has happened?
- No leakage, the model is good
- The test set is faulty
- Overfitting Correct answer
- Underfitting
Explanation: The model has memorised the training data and doesn't generalise.
2 Apply cost-complexity pruning.
m = DecisionTreeClassifier(____=0.02)m = DecisionTreeClassifier(ccp_alpha=0.02)Explanation: The tree shrinks as ccp_alpha grows.
3 Match the situations.
| Term | Meaning |
|---|---|
High training, low test |
Overfitting |
Both low |
Underfitting |
ccp_alpha |
Pruning |
min_samples_leaf |
Prevents tiny leaves |