What is ensemble?

Ensemble combines predictions from multiple models for better results.

Ensemble learning

An ensemble combines the predictions of several models to get a better result than a single model. Random forests and boosting are ensembles; in this group we'll see how to combine models of different kinds.

Voting: voting on predictions or averaging probabilities.
Bagging: copies of the same model on different samples.
Boosting: models that correct errors sequentially.
Stacking: combining predictions with a meta-model.

Error correlation

We can measure how different models are by how much their errors overlap. Models from different families (linear, neighbourhood, tree) usually go wrong in different places:

h1 = m1.predict(X_test) != y_test
h2 = m2.predict(X_test) != y_test
print((h1 & h2).sum(), h1.sum(), h2.sum())   # few shared errors is good

If two models' errors don't overlap at all, voting with a third model can fix most of the errors.

Exercise Error overlap

Train the three models. Print each one's number of test errors, then the number of examples where all three are wrong together.

Show example solution

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=2, stratify=y)
modeller = [("lr", make_pipeline(StandardScaler(), LogisticRegression())), ("knn", make_pipeline(StandardScaler(), KNeighborsClassifier())), ("agac", DecisionTreeClassifier(max_depth=4, random_state=0))]

hata = []
for ad, m in modeller:
    hata.append(m.fit(X_train, y_train).predict(X_test) != y_test)
print([int(h.sum()) for h in hata])
print(int((hata[0] & hata[1] & hata[2]).sum()))

Test yourself

What is the most important condition for an ensemble to beat a single model?
- The models making different errors
- The models being identical
- There being many models
- All of them being bad
Show answer

The models making different errors — Without diversity, combining adds no information.

Count the two models' shared errors.

print((h1 ____ h2).sum())

Show answer

print((h1 & h2).sum())

Match the methods.

voting

Voting or probability averaging
bagging

Sample copies
boosting

Sequential correction
stacking

Meta-model

Next lesson: Voting

We Are Enhancing Your Experience

Cookie Preferences

Strictly Necessary Cookies

Analytical Cookies