Machine Learning
Ensemble
Ensemble methods combine predictions from multiple models.
The ensemble idea
The majority vote of weak predictors making independent errors is more accurate than a single predictor. The majority of 25 independent classifiers, each 60% correct, is right about 84% of the time:
from scipy.stats import binom
print(1 - binom.cdf(12, 25, 0.6)) # 13 or more correct votes: ~0.85
The key word is independent: if they all make the same error, voting gains nothing. The randomness in a random forest is there to increase that independence.
Combining in classification and regression
scikit-learn's RandomForestClassifier averages the trees' probabilities instead of a hard vote (soft voting). RandomForestRegressor gives the mean of the tree predictions:
import numpy as np
ort = np.mean([t.predict_proba(X_test[:1]) for t in rf.estimators_], axis=0)
print(np.allclose(ort, rf.predict_proba(X_test[:1]))) # True
The forest's probabilities are exactly the average of the tree probabilities.
Exercise Forest = mean of trees
Compute the mean of the predict_proba output of every tree of the trained forest for the first test example. Print with np.allclose that it equals rf.predict_proba. Then print the number of trees that say class 1 for the first test example.
Show example solution
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
rf = RandomForestClassifier(n_estimators=50, random_state=0).fit(X_train, y_train)
x = X_test[:1]
ort = np.mean([t.predict_proba(x) for t in rf.estimators_], axis=0)
print(np.allclose(ort, rf.predict_proba(x)))
print(sum(int(t.predict(x)[0] == 1) for t in rf.estimators_))
Test yourself
-
When does voting give the least gain?
- When there are many models
- When every model beats 50%
- When all models make the same errors
- When the models are independent
Show answer
When all models make the same errors — Correlated errors aren't fixed by voting; diversity is needed.
-
Take the mean of the tree probabilities.
ort = np.____([t.predict_proba(x) for t in rf.estimators_], axis=0)
Show answer
ort = np.mean([t.predict_proba(x) for t in rf.estimators_], axis=0)
-
Match the combining methods.
- hard voting
- Majority class
- soft voting
- Probability mean
- regression forest
- Prediction mean
- independent errors
- The power of voting
Next lesson: n_estimators
