What is ensemble?
- Machine Learning course
- Intermediate
- Lesson 97/112
- 5 min
Ensemble combines predictions from multiple models for better results.
Ensemble learning
An ensemble combines the predictions of several models to get a better result than a single model. Random forests and boosting are ensembles; in this group we'll see how to combine models of different kinds.
- Voting: voting on predictions or averaging probabilities.
- Bagging: copies of the same model on different samples.
- Boosting: models that correct errors sequentially.
- Stacking: combining predictions with a meta-model.
Error correlation
We can measure how different models are by how much their errors overlap. Models from different families (linear, neighbourhood, tree) usually go wrong in different places:
h1 = m1.predict(X_test) != y_test
h2 = m2.predict(X_test) != y_test
print((h1 & h2).sum(), h1.sum(), h2.sum()) # few shared errors is goodIf two models' errors don't overlap at all, voting with a third model can fix most of the errors.
Practice Error overlap
Train the three models. Print each one's number of test errors, then the number of examples where all three are wrong together.
Starter code
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=2, stratify=y)
modeller = [("lr", make_pipeline(StandardScaler(), LogisticRegression())), ("knn", make_pipeline(StandardScaler(), KNeighborsClassifier())), ("agac", DecisionTreeClassifier(max_depth=4, random_state=0))]
# buraya yazSample solution
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=2, stratify=y)
modeller = [("lr", make_pipeline(StandardScaler(), LogisticRegression())), ("knn", make_pipeline(StandardScaler(), KNeighborsClassifier())), ("agac", DecisionTreeClassifier(max_depth=4, random_state=0))]
hata = []
for ad, m in modeller:
hata.append(m.fit(X_train, y_train).predict(X_test) != y_test)
print([int(h.sum()) for h in hata])
print(int((hata[0] & hata[1] & hata[2]).sum()))Expected output
[6, 8, 8] 3
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 What is the most important condition for an ensemble to beat a single model?
- The models making different errors Correct answer
- The models being identical
- There being many models
- All of them being bad
Explanation: Without diversity, combining adds no information.
2 Count the two models' shared errors.
print((h1 ____ h2).sum())print((h1 & h2).sum())Explanation: & is element-wise "and": both were wrong.
3 Match the methods.
| Term | Meaning |
|---|---|
voting |
Voting or probability averaging |
bagging |
Sample copies |
boosting |
Sequential correction |
stacking |
Meta-model |