CodeBlanks Logo Icon CodeBlanks
Login Register

Evaluation

Evaluate the model.

Evaluation

The final measurement is done once on the test set. The report shows side by side the baseline (DummyRegressor), the model with the raw features and the model with the derived features:

# baseline: the mean price for every house
# raw:      10 columns
# final:    12 columns (m² per room and log distance added)

This table clearly shows what each step of the project contributed.

Interpreting the coefficients

In a scaled linear model the coefficients are "the price effect of a one-standard-deviation change" and are comparable. The signs must agree with domain knowledge: square metres positive, distance and age negative:

ridge = final["ridge"]
for ad, c in zip(adlar, ridge.coef_):
    print(ad, round(c))

An unexpected sign (e.g. age positive) signals a data error or correlated features.

Practice Final comparison

Compute the test MAE of three models: DummyRegressor, make_pipeline(StandardScaler(), Ridge(alpha=1.0)) with the 10 raw columns, and the final 12-column version (with derived features) of the same chain. Print the three MAEs as integers with round on one line.

Starter code

model.py
import numpy as np

rng = np.random.default_rng(11)
n = 500
m2 = rng.uniform(40, 200, n)
oda = np.clip(np.round(m2 / 35 + rng.normal(0, 0.7, n)), 1, 6)
yas = rng.uniform(0, 40, n)
kat = rng.integers(0, 15, n)
merkez_km = rng.uniform(0.5, 25, n)
gurultu = rng.normal(size=(n, 5))
fiyat = 20 * m2 + 150 * m2 / oda - 2500 * np.log1p(merkez_km) - 40 * yas + 60 * kat + rng.normal(0, 300, n)
X = np.column_stack([m2, oda, yas, kat, merkez_km, gurultu])
X2 = np.column_stack([X, m2 / oda, np.log1p(merkez_km)])
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X2, fiyat, test_size=0.25, random_state=0)
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.feature_selection import SelectKBest, f_regression
from sklearn.linear_model import Ridge
from sklearn.dummy import DummyRegressor
from sklearn.metrics import mean_absolute_error

# buraya yaz

Sample solution

model.py
import numpy as np

rng = np.random.default_rng(11)
n = 500
m2 = rng.uniform(40, 200, n)
oda = np.clip(np.round(m2 / 35 + rng.normal(0, 0.7, n)), 1, 6)
yas = rng.uniform(0, 40, n)
kat = rng.integers(0, 15, n)
merkez_km = rng.uniform(0.5, 25, n)
gurultu = rng.normal(size=(n, 5))
fiyat = 20 * m2 + 150 * m2 / oda - 2500 * np.log1p(merkez_km) - 40 * yas + 60 * kat + rng.normal(0, 300, n)
X = np.column_stack([m2, oda, yas, kat, merkez_km, gurultu])
X2 = np.column_stack([X, m2 / oda, np.log1p(merkez_km)])
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X2, fiyat, test_size=0.25, random_state=0)
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.feature_selection import SelectKBest, f_regression
from sklearn.linear_model import Ridge
from sklearn.dummy import DummyRegressor
from sklearn.metrics import mean_absolute_error

temel = DummyRegressor().fit(X_train, y_train)
ham = make_pipeline(StandardScaler(), Ridge(alpha=1.0)).fit(X_train[:, :10], y_train)
final = make_pipeline(StandardScaler(), Ridge(alpha=1.0)).fit(X_train, y_train)
print(round(mean_absolute_error(y_test, temel.predict(X_test))), round(mean_absolute_error(y_test, ham.predict(X_test[:, :10]))), round(mean_absolute_error(y_test, final.predict(X_test))))

Expected output

2387 852 218

Questions and answers

Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.

1 If adding two derived columns cuts the linear model's MAE by more than half, what can be said?

  • The derived columns are leakage
  • The model has overfitted
  • The test set is faulty
  • The raw columns hid relationships the linear model couldn't see Correct answer

Explanation: Ratio and logarithm relationships can only be learned by a linear model when given as columns.

2 Create the baseline regressor.

Question 
temel = ____().fit(X_train, y_train)
Answer 
temel = DummyRegressor().fit(X_train, y_train)

Explanation: DummyRegressor predicts the mean by default.

3 Match the report rows.

Term Meaning
DummyRegressor Baseline
raw model Before feature work
final model With derived features
coefficient sign Check with domain knowledge

All lessons in Intermediate

1 scikit-learn basics

  1. What is scikit-learn?
  2. Estimator API
  3. The fit method
  4. The predict method
  5. The score method
  6. Summary - scikit-learn

2 Pipeline

  1. What is a Pipeline?
  2. Steps
  3. fit and predict
  4. Advantages
  5. ColumnTransformer
  6. Summary - Pipeline

3 Cross-validation

  1. What is cross-validation?
  2. K-fold
  3. cross_val_score
  4. Stratified
  5. Advantages
  6. Summary - Cross-validation

4 Grid search

  1. What is grid search?
  2. GridSearchCV
  3. param_grid
  4. best_params_
  5. best_score_
  6. Summary - Grid search

5 SVM

  1. What is SVM?
  2. Margin
  3. Kernel
  4. C parameter
  5. Classification
  6. Summary - SVM

6 Project: SVM

  1. Preparing data
  2. Training model
  3. Prediction
  4. Evaluation

7 Random forest

  1. What is random forest?
  2. Ensemble
  3. n_estimators
  4. Bagging
  5. Feature importance
  6. Summary - Random forest

8 Boosting

  1. What is boosting?
  2. AdaBoost
  3. Gradient boosting
  4. Learning rate
  5. Advantages
  6. Summary - Boosting

9 Clustering

  1. What is clustering?
  2. KMeans
  3. Centroid
  4. Elbow method
  5. Labels
  6. Summary - Clustering

10 KMeans in depth

  1. KMeans algorithm
  2. Initialization
  3. n_init
  4. inertia_
  5. random_state
  6. Summary - KMeans

11 Hierarchical clustering

  1. What is hierarchical clustering?
  2. Dendrogram
  3. Agglomerative
  4. Linkage
  5. Advantages
  6. Summary - Hierarchical clustering

12 Project: Clustering

  1. Preparing data
  2. Training KMeans
  3. Label analysis
  4. Printing results

13 PCA

  1. What is PCA?
  2. Variance
  3. n_components
  4. fit_transform
  5. explained_variance_
  6. Summary - PCA

14 Dimensionality reduction

  1. What is dimensionality reduction?
  2. Why reduce?
  3. PCA vs t-SNE
  4. Visualization
  5. Advantages
  6. Summary - Dimensionality reduction

15 Feature selection

  1. What is feature selection?
  2. Variance threshold
  3. SelectKBest
  4. Recursive elimination
  5. Feature importance
  6. Summary - Feature selection

16 Imbalanced data

  1. What is imbalanced data?
  2. SMOTE
  3. Class weight
  4. Oversampling
  5. Metric selection
  6. Summary - Imbalanced data

17 Project: Features

  1. Feature extraction
  2. Feature selection
  3. Training model
  4. Evaluation

18 Ensemble methods

  1. What is ensemble?
  2. Voting
  3. Bagging
  4. Boosting
  5. Stacking
  6. Summary - Ensemble

19 Regularization

  1. What is regularization?
  2. L1 (Lasso)
  3. L2 (Ridge)
  4. alpha
  5. ElasticNet
  6. Summary - Regularization

20 Intermediate final

  1. Challenge 1
  2. Challenge 2
  3. Challenge 3
  4. Challenge 4