Voting
- Machine Learning course
- Intermediate
- Lesson 98/112
- 5 min
Voting ensemble lets models decide by majority vote.
VotingClassifier
VotingClassifier makes different models a single estimator. It has two forms:
voting="hard": each model casts a class vote, the majority wins.voting="soft": the probabilities are averaged; how confident the models are is taken into account; usually better.
from sklearn.ensemble import VotingClassifier
oy = VotingClassifier(modeller, voting="soft").fit(X_train, y_train)
print(oy.score(X_test, y_test))Soft voting requires all models to offer predict_proba.
Weighted voting
The weights parameter gives more reliable models a bigger say. The weights are hyperparameters too and should be chosen with CV:
oy = VotingClassifier(modeller, voting="soft", weights=[2, 1, 1])
# lr's probability counts twiceIf one of the ensemble's models is clearly bad, removing it is usually better than giving it a low weight.
Practice Hard and soft
Train a VotingClassifier with the three models first with voting="hard", then with voting="soft". Print both test accuracies with round(..., 3) on one line.
Starter code
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=2, stratify=y)
modeller = [("lr", make_pipeline(StandardScaler(), LogisticRegression())), ("knn", make_pipeline(StandardScaler(), KNeighborsClassifier())), ("agac", DecisionTreeClassifier(max_depth=4, random_state=0))]
from sklearn.ensemble import VotingClassifier
# buraya yazSample solution
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=2, stratify=y)
modeller = [("lr", make_pipeline(StandardScaler(), LogisticRegression())), ("knn", make_pipeline(StandardScaler(), KNeighborsClassifier())), ("agac", DecisionTreeClassifier(max_depth=4, random_state=0))]
from sklearn.ensemble import VotingClassifier
sonuc = []
for v in ["hard", "soft"]:
oy = VotingClassifier(modeller, voting=v).fit(X_train, y_train)
sonuc.append(round(oy.score(X_test, y_test), 3))
print(*sonuc)Expected output
0.965 0.971
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 If the probabilities (class 1) are [0.9, 0.4, 0.45], what do hard and soft voting decide?
- Both 1
- Both 0
- Hard: 1, soft: 0
- Hard: 0, soft: 1 Correct answer
Explanation: Hard: two votes for 0 (0.4, 0.45 < 0.5). Soft: the mean 0.583 > 0.5, class 1.
2 Vote by averaging probabilities.
oy = VotingClassifier(modeller, voting=____)oy = VotingClassifier(modeller, voting="soft")Explanation: "soft" uses the mean of predict_proba.
3 Match the settings.
| Term | Meaning |
|---|---|
voting="hard" |
Majority class |
voting="soft" |
Probability mean |
weights |
Model weights |
estimators |
List of (name, model) |