Ensemble
- Machine Learning course
- Intermediate
- Lesson 36/112
- 5 min
Ensemble methods combine predictions from multiple models.
The ensemble idea
The majority vote of weak predictors making independent errors is more accurate than a single predictor. The majority of 25 independent classifiers, each 60% correct, is right about 84% of the time:
from scipy.stats import binom
print(1 - binom.cdf(12, 25, 0.6)) # 13 or more correct votes: ~0.85The key word is independent: if they all make the same error, voting gains nothing. The randomness in a random forest is there to increase that independence.
Combining in classification and regression
scikit-learn's RandomForestClassifier averages the trees' probabilities instead of a hard vote (soft voting). RandomForestRegressor gives the mean of the tree predictions:
import numpy as np
ort = np.mean([t.predict_proba(X_test[:1]) for t in rf.estimators_], axis=0)
print(np.allclose(ort, rf.predict_proba(X_test[:1]))) # TrueThe forest's probabilities are exactly the average of the tree probabilities.
Practice Forest = mean of trees
Compute the mean of the predict_proba output of every tree of the trained forest for the first test example. Print with np.allclose that it equals rf.predict_proba. Then print the number of trees that say class 1 for the first test example.
Starter code
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
rf = RandomForestClassifier(n_estimators=50, random_state=0).fit(X_train, y_train)
# buraya yazSample solution
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
rf = RandomForestClassifier(n_estimators=50, random_state=0).fit(X_train, y_train)
x = X_test[:1]
ort = np.mean([t.predict_proba(x) for t in rf.estimators_], axis=0)
print(np.allclose(ort, rf.predict_proba(x)))
print(sum(int(t.predict(x)[0] == 1) for t in rf.estimators_))Expected output
True 48
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 When does voting give the least gain?
- When there are many models
- When every model beats 50%
- When all models make the same errors Correct answer
- When the models are independent
Explanation: Correlated errors aren't fixed by voting; diversity is needed.
2 Take the mean of the tree probabilities.
ort = np.____([t.predict_proba(x) for t in rf.estimators_], axis=0)ort = np.mean([t.predict_proba(x) for t in rf.estimators_], axis=0)Explanation: The forest uses the mean of the trees' probabilities.
3 Match the combining methods.
| Term | Meaning |
|---|---|
hard voting |
Majority class |
soft voting |
Probability mean |
regression forest |
Prediction mean |
independent errors |
The power of voting |