Stacking
- Machine Learning course
- Intermediate
- Lesson 101/112
- 5 min
Stacking combines model predictions with a meta-model.
Stacking
Stacking trains a meta-model that takes the base models' predictions as input. The meta-model learns which base model to trust when. To prevent leakage, the meta-model is trained on the base models' out-of-fold predictions:
from sklearn.ensemble import StackingClassifier
st = StackingClassifier(modeller, final_estimator=LogisticRegression(), cv=5)
st.fit(X_train, y_train)
print(st.score(X_test, y_test))cv=5 makes the base models' predictions for the meta-model be taken internally with 5-fold CV.
Building stacking by hand
We can see the logic explicitly with cross_val_predict: each base model's out-of-fold probabilities become new features, and the meta-model is trained on them:
Z = np.column_stack([
cross_val_predict(m, X_train, y_train, cv=5, method="predict_proba")[:, 1]
for _, m in modeller])
ust = LogisticRegression().fit(Z, y_train)
print(ust.coef_) # how much each model is trustedWith passthrough=True the meta-model also sees the original features.
Practice Stacking
Train a StackingClassifier(..., cv=5) with the three base models and a LogisticRegression() meta-model. Print the test accuracy with round(..., 3), then the meta-model's coefficients with np.round(st.final_estimator_.coef_, 2).
Starter code
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=2, stratify=y)
modeller = [("lr", make_pipeline(StandardScaler(), LogisticRegression())), ("knn", make_pipeline(StandardScaler(), KNeighborsClassifier())), ("agac", DecisionTreeClassifier(max_depth=4, random_state=0))]
from sklearn.ensemble import StackingClassifier
# buraya yazSample solution
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=2, stratify=y)
modeller = [("lr", make_pipeline(StandardScaler(), LogisticRegression())), ("knn", make_pipeline(StandardScaler(), KNeighborsClassifier())), ("agac", DecisionTreeClassifier(max_depth=4, random_state=0))]
from sklearn.ensemble import StackingClassifier
st = StackingClassifier(modeller, final_estimator=LogisticRegression(), cv=5).fit(X_train, y_train)
print(round(st.score(X_test, y_test), 3))
print(np.round(st.final_estimator_.coef_, 2))Expected output
0.971 [[4.06 2.74 1.07]]
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 Why isn't the meta-model trained on the base models' predictions on the training data?
- Those predictions are over-optimistic; the meta-model learns to trust the memorising model Correct answer
- They can't be computed
- It is too slow
- It is trained on them
Explanation: Out-of-fold predictions reflect the base models' real behaviour on new data.
2 Get the out-of-fold probabilities.
p = cross_val_predict(m, X_train, y_train, cv=5, method=____)p = cross_val_predict(m, X_train, y_train, cv=5, method="predict_proba")Explanation: method selects which method cross_val_predict calls.
3 Match the pieces.
| Term | Meaning |
|---|---|
base models |
First level |
final_estimator |
Meta-model |
out-of-fold prediction |
Leak-free input |
passthrough |
Also add original features |