Machine Learning
Challenge 3
Train a regression model and make predictions.
Challenge 3: catch the leakage
A teammate reports near-100% accuracy with this code. The problem is that scaling is done on all data before the split and, worse, a feature derived from the label is added to X:
X2 = np.column_stack([X, y + rng.normal(0, 0.1, len(y))]) # label leaks!
X2 = StandardScaler().fit_transform(X2) # fit on all data
X_tr, X_te, y_tr, y_te = train_test_split(X2, y, ...)
In real life leakage is usually not this obvious: columns that indirectly know the target, like "transaction cancellation date" or "inspection result".
Exercise Show the leakage
Train a make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)) on the same split with the leaky X2 and with the original X. Print the two test accuracies with round(..., 3) on separate lines (leaky first).
Show example solution
import numpy as np
from sklearn.datasets import load_wine
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
X, y = load_wine(return_X_y=True)
rng = np.random.default_rng(0)
X = X[:, :2]
X2 = np.column_stack([X, y + rng.normal(0, 0.1, len(y))])
for veri in [X2, X]:
X_tr, X_te, y_tr, y_te = train_test_split(veri, y, test_size=0.3, random_state=0, stratify=y)
m = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)).fit(X_tr, y_tr)
print(round(m.score(X_te, y_te), 3))
Exercise Find the suspicious column
For a DecisionTreeClassifier(max_depth=3, random_state=0) using each column of X2 on its own, compute the mean 5-fold CV score. Print the scores with np.round(..., 2), then the index of the highest-scoring column.
Show example solution
import numpy as np
from sklearn.datasets import load_wine
from sklearn.model_selection import cross_val_score
from sklearn.tree import DecisionTreeClassifier
X, y = load_wine(return_X_y=True)
rng = np.random.default_rng(0)
X2 = np.column_stack([X[:, :2], y + rng.normal(0, 0.1, len(y))])
skor = [cross_val_score(DecisionTreeClassifier(max_depth=3, random_state=0), X2[:, [j]], y, cv=5).mean() for j in range(X2.shape[1])]
print(np.round(skor, 2))
print(int(np.argmax(skor)))
Test yourself
-
What is the strongest sign to suspect a feature of leakage?
- It contains missing values
- It is categorical
- It isn't scaled
- It gives near-perfect predictions on its own
Show answer
It gives near-perfect predictions on its own — Real problems are rarely solved by a single feature; a perfect result usually means the target has leaked.
-
Put the scaling inside the pipeline.
model = make_pipeline(____, LogisticRegression())
Show answer
model = make_pipeline(StandardScaler(), LogisticRegression())
-
Match the mistakes with their fixes.
- Scaling on all the data
- Use a pipeline
- Column derived from the label
- Drop the column
- Choosing the model on the test set
- Choose with CV
- Shuffling a time series
- shuffle=False
Next lesson: Challenge 4
