Machine Learning
Feature importance
Random forest computes feature importance.
Feature importance (impurity)
feature_importances_ is the average of how much each feature reduces impurity in the splits across all trees. It is fast but has two weaknesses: it favours features with many distinct values (high cardinality) and relies on the training data.
d = load_breast_cancer()
rf.fit(d.data, d.target)
sira = np.argsort(rf.feature_importances_)[::-1][:3]
for i in sira:
print(d.feature_names[i], round(rf.feature_importances_[i], 3))
The importances sum to 1; correlated features share importance between them.
Permutation importance
The more reliable method: shuffle a column's values (breaking its relationship) and see how much the test score drops. If the drop is large, the model really depends on that feature:
from sklearn.inspection import permutation_importance
r = permutation_importance(rf, X_test, y_test, n_repeats=10, random_state=0)
print(r.importances_mean.round(3))
Because it is computed on test data, it doesn't show memorised noise as important; it works with any model type.
Exercise A noise column
Add a column of random numbers to the data (the last column). Train a forest; print the noise column's impurity importance and permutation importance (n_repeats=5, random_state=0, on the test data) with round(..., 3) on one line.
Show example solution
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.inspection import permutation_importance
X, y = load_breast_cancer(return_X_y=True)
rng = np.random.default_rng(0)
X = np.column_stack([X[:, :5], rng.random(len(X))])
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
rf = RandomForestClassifier(n_estimators=200, random_state=0).fit(X_train, y_train)
r = permutation_importance(rf, X_test, y_test, n_repeats=5, random_state=0)
print(round(rf.feature_importances_[-1], 3), round(r.importances_mean[-1], 3))
Test yourself
-
How does a meaningless column with many distinct values, like a random ID, affect impurity importance?
- Its importance is always zero
- It errors
- It raises the other importances
- It can get artificially high importance
Show answer
It can get artificially high importance — A many-valued column gives trees many split opportunities; it separates noise in training.
-
Compute the permutation importance.
r = ____(rf, X_test, y_test, n_repeats=10, random_state=0)
Show answer
r = permutation_importance(rf, X_test, y_test, n_repeats=10, random_state=0)
-
Match the methods.
- feature_importances_
- Impurity decrease
- permutation_importance
- Score drop when shuffled
- high cardinality
- Inflates impurity importance
- test data
- Permutation's safeguard
Next lesson: Summary - Random forest
