Challenge 3
- Machine Learning course
- Beginner
- Lesson 111/112
- 8 min
Train a regression model and make predictions.
Challenge 3: catch the leakage
A teammate reports near-100% accuracy with this code. The problem is that scaling is done on all data before the split and, worse, a feature derived from the label is added to X:
X2 = np.column_stack([X, y + rng.normal(0, 0.1, len(y))]) # label leaks!
X2 = StandardScaler().fit_transform(X2) # fit on all data
X_tr, X_te, y_tr, y_te = train_test_split(X2, y, ...)In real life leakage is usually not this obvious: columns that indirectly know the target, like "transaction cancellation date" or "inspection result".
Practice Show the leakage
Train a make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)) on the same split with the leaky X2 and with the original X. Print the two test accuracies with round(..., 3) on separate lines (leaky first).
Starter code
import numpy as np
from sklearn.datasets import load_wine
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
X, y = load_wine(return_X_y=True)
rng = np.random.default_rng(0)
X = X[:, :2]
X2 = np.column_stack([X, y + rng.normal(0, 0.1, len(y))])
# buraya yazSample solution
import numpy as np
from sklearn.datasets import load_wine
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
X, y = load_wine(return_X_y=True)
rng = np.random.default_rng(0)
X = X[:, :2]
X2 = np.column_stack([X, y + rng.normal(0, 0.1, len(y))])
for veri in [X2, X]:
X_tr, X_te, y_tr, y_te = train_test_split(veri, y, test_size=0.3, random_state=0, stratify=y)
m = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)).fit(X_tr, y_tr)
print(round(m.score(X_te, y_te), 3))Expected output
1.0 0.778
Practice Find the suspicious column
For a DecisionTreeClassifier(max_depth=3, random_state=0) using each column of X2 on its own, compute the mean 5-fold CV score. Print the scores with np.round(..., 2), then the index of the highest-scoring column.
Starter code
import numpy as np
from sklearn.datasets import load_wine
from sklearn.model_selection import cross_val_score
from sklearn.tree import DecisionTreeClassifier
X, y = load_wine(return_X_y=True)
rng = np.random.default_rng(0)
X2 = np.column_stack([X[:, :2], y + rng.normal(0, 0.1, len(y))])
# buraya yazSample solution
import numpy as np
from sklearn.datasets import load_wine
from sklearn.model_selection import cross_val_score
from sklearn.tree import DecisionTreeClassifier
X, y = load_wine(return_X_y=True)
rng = np.random.default_rng(0)
X2 = np.column_stack([X[:, :2], y + rng.normal(0, 0.1, len(y))])
skor = [cross_val_score(DecisionTreeClassifier(max_depth=3, random_state=0), X2[:, [j]], y, cv=5).mean() for j in range(X2.shape[1])]
print(np.round(skor, 2))
print(int(np.argmax(skor)))Expected output
[0.64 0.62 1. ] 2
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 What is the strongest sign to suspect a feature of leakage?
- It contains missing values
- It is categorical
- It isn't scaled
- It gives near-perfect predictions on its own Correct answer
Explanation: Real problems are rarely solved by a single feature; a perfect result usually means the target has leaked.
2 Put the scaling inside the pipeline.
model = make_pipeline(____, LogisticRegression())model = make_pipeline(StandardScaler(), LogisticRegression())Explanation: A scaler inside a pipeline is fit only on training data.
3 Match the mistakes with their fixes.
| Term | Meaning |
|---|---|
Scaling on all the data |
Use a pipeline |
Column derived from the label |
Drop the column |
Choosing the model on the test set |
Choose with CV |
Shuffling a time series |
shuffle=False |