Summary - Ensemble

In this group you learned voting, bagging, boosting, and stacking.

Summary: ensembles

In this group:

An ensemble combines models making different errors; diversity is a must.
VotingClassifier: hard (majority) and soft (probability averaging), weights.
BaggingClassifier: reduces the variance of unstable models.
Boosting reduces bias; for stumps boosting helps, not bagging.
StackingClassifier: a meta-model on out-of-fold predictions.

Ensembles usually gain a point or two but are slower and harder to interpret; whether that price is worth it depends on the problem.

Exercise Compare the ensembles

Compare the three single models, a soft VotingClassifier and a StackingClassifier (logistic meta-model, cv=5) with 5-fold CV. On each line print the name and the mean with round(..., 3) (in order: lr, knn, agac, oylama, yigin).

Show example solution

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=2, stratify=y)
modeller = [("lr", make_pipeline(StandardScaler(), LogisticRegression())), ("knn", make_pipeline(StandardScaler(), KNeighborsClassifier())), ("agac", DecisionTreeClassifier(max_depth=4, random_state=0))]
from sklearn.ensemble import VotingClassifier, StackingClassifier
from sklearn.model_selection import cross_val_score

aday = modeller + [("oylama", VotingClassifier(modeller, voting="soft")), ("yigin", StackingClassifier(modeller, final_estimator=LogisticRegression(), cv=5))]
for ad, m in aday:
    print(ad, round(cross_val_score(m, X, y, cv=5).mean(), 3))

Test yourself

What is required for soft voting?
- All models being the same type
- An odd number of models
- Unscaled data
- All models offering predict_proba
Show answer

All models offering predict_proba — Averaging probabilities requires every model to produce probabilities.

Let the meta-model also see the original features.

st = StackingClassifier(modeller, final_estimator=LogisticRegression(), ____=True)

Show answer

st = StackingClassifier(modeller, final_estimator=LogisticRegression(), passthrough=True)

Match the tools.

VotingClassifier

Voting
BaggingClassifier

Bootstrap copies
AdaBoostClassifier

Sequential reweighting
StackingClassifier

Meta-model

Next lesson: What is regularization?

We Are Enhancing Your Experience

Cookie Preferences

Strictly Necessary Cookies

Analytical Cookies