Machine Learning
Stacking
Stacking combines model predictions with a meta-model.
Stacking
Stacking trains a meta-model that takes the base models' predictions as input. The meta-model learns which base model to trust when. To prevent leakage, the meta-model is trained on the base models' out-of-fold predictions:
from sklearn.ensemble import StackingClassifier
st = StackingClassifier(modeller, final_estimator=LogisticRegression(), cv=5)
st.fit(X_train, y_train)
print(st.score(X_test, y_test))
cv=5 makes the base models' predictions for the meta-model be taken internally with 5-fold CV.
Building stacking by hand
We can see the logic explicitly with cross_val_predict: each base model's out-of-fold probabilities become new features, and the meta-model is trained on them:
Z = np.column_stack([
cross_val_predict(m, X_train, y_train, cv=5, method="predict_proba")[:, 1]
for _, m in modeller])
ust = LogisticRegression().fit(Z, y_train)
print(ust.coef_) # how much each model is trusted
With passthrough=True the meta-model also sees the original features.
Exercise Stacking
Train a StackingClassifier(..., cv=5) with the three base models and a LogisticRegression() meta-model. Print the test accuracy with round(..., 3), then the meta-model's coefficients with np.round(st.final_estimator_.coef_, 2).
Show example solution
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=2, stratify=y)
modeller = [("lr", make_pipeline(StandardScaler(), LogisticRegression())), ("knn", make_pipeline(StandardScaler(), KNeighborsClassifier())), ("agac", DecisionTreeClassifier(max_depth=4, random_state=0))]
from sklearn.ensemble import StackingClassifier
st = StackingClassifier(modeller, final_estimator=LogisticRegression(), cv=5).fit(X_train, y_train)
print(round(st.score(X_test, y_test), 3))
print(np.round(st.final_estimator_.coef_, 2))
Test yourself
-
Why isn't the meta-model trained on the base models' predictions on the training data?
- Those predictions are over-optimistic; the meta-model learns to trust the memorising model
- They can't be computed
- It is too slow
- It is trained on them
Show answer
Those predictions are over-optimistic; the meta-model learns to trust the memorising model — Out-of-fold predictions reflect the base models' real behaviour on new data.
-
Get the out-of-fold probabilities.
p = cross_val_predict(m, X_train, y_train, cv=5, method=____)
Show answer
p = cross_val_predict(m, X_train, y_train, cv=5, method="predict_proba")
-
Match the pieces.
- base models
- First level
- final_estimator
- Meta-model
- out-of-fold prediction
- Leak-free input
- passthrough
- Also add original features
Next lesson: Summary - Ensemble
