Summary - Ensemble
- Machine Learning course
- Intermediate
- Lesson 102/112
- 6 min
In this group you learned voting, bagging, boosting, and stacking.
Summary: ensembles
In this group:
- An ensemble combines models making different errors; diversity is a must.
VotingClassifier: hard (majority) and soft (probability averaging),weights.BaggingClassifier: reduces the variance of unstable models.- Boosting reduces bias; for stumps boosting helps, not bagging.
StackingClassifier: a meta-model on out-of-fold predictions.
Ensembles usually gain a point or two but are slower and harder to interpret; whether that price is worth it depends on the problem.
Practice Compare the ensembles
Compare the three single models, a soft VotingClassifier and a StackingClassifier (logistic meta-model, cv=5) with 5-fold CV. On each line print the name and the mean with round(..., 3) (in order: lr, knn, agac, oylama, yigin).
Starter code
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=2, stratify=y)
modeller = [("lr", make_pipeline(StandardScaler(), LogisticRegression())), ("knn", make_pipeline(StandardScaler(), KNeighborsClassifier())), ("agac", DecisionTreeClassifier(max_depth=4, random_state=0))]
from sklearn.ensemble import VotingClassifier, StackingClassifier
from sklearn.model_selection import cross_val_score
# buraya yazSample solution
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=2, stratify=y)
modeller = [("lr", make_pipeline(StandardScaler(), LogisticRegression())), ("knn", make_pipeline(StandardScaler(), KNeighborsClassifier())), ("agac", DecisionTreeClassifier(max_depth=4, random_state=0))]
from sklearn.ensemble import VotingClassifier, StackingClassifier
from sklearn.model_selection import cross_val_score
aday = modeller + [("oylama", VotingClassifier(modeller, voting="soft")), ("yigin", StackingClassifier(modeller, final_estimator=LogisticRegression(), cv=5))]
for ad, m in aday:
print(ad, round(cross_val_score(m, X, y, cv=5).mean(), 3))Expected output
lr 0.981 knn 0.965 agac 0.916 oylama 0.958 yigin 0.977
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 What is required for soft voting?
- All models being the same type
- An odd number of models
- Unscaled data
- All models offering predict_proba Correct answer
Explanation: Averaging probabilities requires every model to produce probabilities.
2 Let the meta-model also see the original features.
Question
st = StackingClassifier(modeller, final_estimator=LogisticRegression(), ____=True)
Answer
st = StackingClassifier(modeller, final_estimator=LogisticRegression(), passthrough=True)Explanation: passthrough=True appends the original X next to the base predictions.
3 Match the tools.
| Term | Meaning |
|---|---|
VotingClassifier |
Voting |
BaggingClassifier |
Bootstrap copies |
AdaBoostClassifier |
Sequential reweighting |
StackingClassifier |
Meta-model |