Machine Learning
Voting
Voting ensemble lets models decide by majority vote.
VotingClassifier
VotingClassifier makes different models a single estimator. It has two forms:
voting="hard": each model casts a class vote, the majority wins.voting="soft": the probabilities are averaged; how confident the models are is taken into account; usually better.
from sklearn.ensemble import VotingClassifier
oy = VotingClassifier(modeller, voting="soft").fit(X_train, y_train)
print(oy.score(X_test, y_test))
Soft voting requires all models to offer predict_proba.
Weighted voting
The weights parameter gives more reliable models a bigger say. The weights are hyperparameters too and should be chosen with CV:
oy = VotingClassifier(modeller, voting="soft", weights=[2, 1, 1])
# lr's probability counts twice
If one of the ensemble's models is clearly bad, removing it is usually better than giving it a low weight.
Exercise Hard and soft
Train a VotingClassifier with the three models first with voting="hard", then with voting="soft". Print both test accuracies with round(..., 3) on one line.
Show example solution
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=2, stratify=y)
modeller = [("lr", make_pipeline(StandardScaler(), LogisticRegression())), ("knn", make_pipeline(StandardScaler(), KNeighborsClassifier())), ("agac", DecisionTreeClassifier(max_depth=4, random_state=0))]
from sklearn.ensemble import VotingClassifier
sonuc = []
for v in ["hard", "soft"]:
oy = VotingClassifier(modeller, voting=v).fit(X_train, y_train)
sonuc.append(round(oy.score(X_test, y_test), 3))
print(*sonuc)
Test yourself
-
If the probabilities (class 1) are [0.9, 0.4, 0.45], what do hard and soft voting decide?
- Both 1
- Both 0
- Hard: 1, soft: 0
- Hard: 0, soft: 1
Show answer
Hard: 0, soft: 1 — Hard: two votes for 0 (0.4, 0.45 < 0.5). Soft: the mean 0.583 > 0.5, class 1.
-
Vote by averaging probabilities.
oy = VotingClassifier(modeller, voting=____)
Show answer
oy = VotingClassifier(modeller, voting="soft")
-
Match the settings.
- voting="hard"
- Majority class
- voting="soft"
- Probability mean
- weights
- Model weights
- estimators
- List of (name, model)
Next lesson: Bagging
