Machine Learning
Evaluation
Evaluate the model.
Evaluation
The final measurement is done once on the test set. The report shows side by side the baseline (DummyRegressor), the model with the raw features and the model with the derived features:
# baseline: the mean price for every house
# raw: 10 columns
# final: 12 columns (m² per room and log distance added)
This table clearly shows what each step of the project contributed.
Interpreting the coefficients
In a scaled linear model the coefficients are "the price effect of a one-standard-deviation change" and are comparable. The signs must agree with domain knowledge: square metres positive, distance and age negative:
ridge = final["ridge"]
for ad, c in zip(adlar, ridge.coef_):
print(ad, round(c))
An unexpected sign (e.g. age positive) signals a data error or correlated features.
Exercise Final comparison
Compute the test MAE of three models: DummyRegressor, make_pipeline(StandardScaler(), Ridge(alpha=1.0)) with the 10 raw columns, and the final 12-column version (with derived features) of the same chain. Print the three MAEs as integers with round on one line.
Show example solution
import numpy as np
rng = np.random.default_rng(11)
n = 500
m2 = rng.uniform(40, 200, n)
oda = np.clip(np.round(m2 / 35 + rng.normal(0, 0.7, n)), 1, 6)
yas = rng.uniform(0, 40, n)
kat = rng.integers(0, 15, n)
merkez_km = rng.uniform(0.5, 25, n)
gurultu = rng.normal(size=(n, 5))
fiyat = 20 * m2 + 150 * m2 / oda - 2500 * np.log1p(merkez_km) - 40 * yas + 60 * kat + rng.normal(0, 300, n)
X = np.column_stack([m2, oda, yas, kat, merkez_km, gurultu])
X2 = np.column_stack([X, m2 / oda, np.log1p(merkez_km)])
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X2, fiyat, test_size=0.25, random_state=0)
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.feature_selection import SelectKBest, f_regression
from sklearn.linear_model import Ridge
from sklearn.dummy import DummyRegressor
from sklearn.metrics import mean_absolute_error
temel = DummyRegressor().fit(X_train, y_train)
ham = make_pipeline(StandardScaler(), Ridge(alpha=1.0)).fit(X_train[:, :10], y_train)
final = make_pipeline(StandardScaler(), Ridge(alpha=1.0)).fit(X_train, y_train)
print(round(mean_absolute_error(y_test, temel.predict(X_test))), round(mean_absolute_error(y_test, ham.predict(X_test[:, :10]))), round(mean_absolute_error(y_test, final.predict(X_test))))
Test yourself
-
If adding two derived columns cuts the linear model's MAE by more than half, what can be said?
- The derived columns are leakage
- The model has overfitted
- The test set is faulty
- The raw columns hid relationships the linear model couldn't see
Show answer
The raw columns hid relationships the linear model couldn't see — Ratio and logarithm relationships can only be learned by a linear model when given as columns.
-
Create the baseline regressor.
temel = ____().fit(X_train, y_train)
Show answer
temel = DummyRegressor().fit(X_train, y_train)
-
Match the report rows.
- DummyRegressor
- Baseline
- raw model
- Before feature work
- final model
- With derived features
- coefficient sign
- Check with domain knowledge
Next lesson: What is ensemble?
