Feature importance
- Machine Learning course
- Intermediate
- Lesson 39/112
- 5 min
Random forest computes feature importance.
Feature importance (impurity)
feature_importances_ is the average of how much each feature reduces impurity in the splits across all trees. It is fast but has two weaknesses: it favours features with many distinct values (high cardinality) and relies on the training data.
d = load_breast_cancer()
rf.fit(d.data, d.target)
sira = np.argsort(rf.feature_importances_)[::-1][:3]
for i in sira:
print(d.feature_names[i], round(rf.feature_importances_[i], 3))The importances sum to 1; correlated features share importance between them.
Permutation importance
The more reliable method: shuffle a column's values (breaking its relationship) and see how much the test score drops. If the drop is large, the model really depends on that feature:
from sklearn.inspection import permutation_importance
r = permutation_importance(rf, X_test, y_test, n_repeats=10, random_state=0)
print(r.importances_mean.round(3))Because it is computed on test data, it doesn't show memorised noise as important; it works with any model type.
Practice A noise column
Add a column of random numbers to the data (the last column). Train a forest; print the noise column's impurity importance and permutation importance (n_repeats=5, random_state=0, on the test data) with round(..., 3) on one line.
Starter code
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.inspection import permutation_importance
X, y = load_breast_cancer(return_X_y=True)
rng = np.random.default_rng(0)
X = np.column_stack([X[:, :5], rng.random(len(X))])
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
# buraya yazSample solution
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.inspection import permutation_importance
X, y = load_breast_cancer(return_X_y=True)
rng = np.random.default_rng(0)
X = np.column_stack([X[:, :5], rng.random(len(X))])
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
rf = RandomForestClassifier(n_estimators=200, random_state=0).fit(X_train, y_train)
r = permutation_importance(rf, X_test, y_test, n_repeats=5, random_state=0)
print(round(rf.feature_importances_[-1], 3), round(r.importances_mean[-1], 3))Expected output
0.035 -0.014
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 How does a meaningless column with many distinct values, like a random ID, affect impurity importance?
- Its importance is always zero
- It errors
- It raises the other importances
- It can get artificially high importance Correct answer
Explanation: A many-valued column gives trees many split opportunities; it separates noise in training.
2 Compute the permutation importance.
r = ____(rf, X_test, y_test, n_repeats=10, random_state=0)r = permutation_importance(rf, X_test, y_test, n_repeats=10, random_state=0)Explanation: permutation_importance lives in sklearn.inspection.
3 Match the methods.
| Term | Meaning |
|---|---|
feature_importances_ |
Impurity decrease |
permutation_importance |
Score drop when shuffled |
high cardinality |
Inflates impurity importance |
test data |
Permutation's safeguard |