CodeBlanks Logo Icon CodeBlanks
Login Register

Feature extraction

Derive new features.

Project: house prices

In this project we'll predict the prices of 500 houses. Features: square metres, room count, building age, floor, distance to the centre and five meaningless noise columns. The project's focus is feature engineering and selection.

X = np.column_stack([m2, oda, yas, kat, merkez_km, gurultu])
print(X.shape)   # (500, 10)

In real projects nobody knows which column is noise; here we added them on purpose so we can see whether the selection methods find them.

Feature extraction

We derive two new features from domain knowledge: square metres per room and the logarithm of the distance to the centre (the first kilometres make more difference than the last ones):

m2_oda = m2 / oda
log_merkez = np.log1p(merkez_km)
X2 = np.column_stack([X, m2_oda, log_merkez])
print(X2.shape)   # (500, 12)

Whether derived features help is again measured by validation; intuition is a hypothesis, not a result.

Practice Derive the features

Build X2 by adding the columns m2 / oda and np.log1p(merkez_km) to X. Print X2.shape, then the means of the two new columns with np.round(..., 2).

Starter code

model.py
import numpy as np

rng = np.random.default_rng(11)
n = 500
m2 = rng.uniform(40, 200, n)
oda = np.clip(np.round(m2 / 35 + rng.normal(0, 0.7, n)), 1, 6)
yas = rng.uniform(0, 40, n)
kat = rng.integers(0, 15, n)
merkez_km = rng.uniform(0.5, 25, n)
gurultu = rng.normal(size=(n, 5))
fiyat = 20 * m2 + 150 * m2 / oda - 2500 * np.log1p(merkez_km) - 40 * yas + 60 * kat + rng.normal(0, 300, n)
X = np.column_stack([m2, oda, yas, kat, merkez_km, gurultu])

# buraya yaz

Sample solution

model.py
import numpy as np

rng = np.random.default_rng(11)
n = 500
m2 = rng.uniform(40, 200, n)
oda = np.clip(np.round(m2 / 35 + rng.normal(0, 0.7, n)), 1, 6)
yas = rng.uniform(0, 40, n)
kat = rng.integers(0, 15, n)
merkez_km = rng.uniform(0.5, 25, n)
gurultu = rng.normal(size=(n, 5))
fiyat = 20 * m2 + 150 * m2 / oda - 2500 * np.log1p(merkez_km) - 40 * yas + 60 * kat + rng.normal(0, 300, n)
X = np.column_stack([m2, oda, yas, kat, merkez_km, gurultu])

X2 = np.column_stack([X, m2 / oda, np.log1p(merkez_km)])
print(X2.shape)
print(np.round(X2[:, -2:].mean(axis=0), 2))

Expected output

(500, 12)
[37.7   2.43]

Questions and answers

Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.

1 Why might square metres per room be a useful derived feature?

  • It increases the row count
  • It is required
  • It describes how spacious the house is in one number Correct answer
  • It reduces noise

Explanation: Fewer rooms at the same area mean larger rooms; the two raw columns don't say this directly.

2 Add the new columns to the feature matrix.

Question 
X2 = np.____([X, m2_oda, log_merkez])
Answer 
X2 = np.column_stack([X, m2_oda, log_merkez])

Explanation: column_stack adds one-dimensional arrays as columns.

3 Match the features.

Term Meaning
m2 Raw area
m2 / oda Spaciousness
log1p(merkez_km) Distance with diminishing effect
gurultu Meaningless columns

All lessons in Intermediate

1 scikit-learn basics

  1. What is scikit-learn?
  2. Estimator API
  3. The fit method
  4. The predict method
  5. The score method
  6. Summary - scikit-learn

2 Pipeline

  1. What is a Pipeline?
  2. Steps
  3. fit and predict
  4. Advantages
  5. ColumnTransformer
  6. Summary - Pipeline

3 Cross-validation

  1. What is cross-validation?
  2. K-fold
  3. cross_val_score
  4. Stratified
  5. Advantages
  6. Summary - Cross-validation

4 Grid search

  1. What is grid search?
  2. GridSearchCV
  3. param_grid
  4. best_params_
  5. best_score_
  6. Summary - Grid search

5 SVM

  1. What is SVM?
  2. Margin
  3. Kernel
  4. C parameter
  5. Classification
  6. Summary - SVM

6 Project: SVM

  1. Preparing data
  2. Training model
  3. Prediction
  4. Evaluation

7 Random forest

  1. What is random forest?
  2. Ensemble
  3. n_estimators
  4. Bagging
  5. Feature importance
  6. Summary - Random forest

8 Boosting

  1. What is boosting?
  2. AdaBoost
  3. Gradient boosting
  4. Learning rate
  5. Advantages
  6. Summary - Boosting

9 Clustering

  1. What is clustering?
  2. KMeans
  3. Centroid
  4. Elbow method
  5. Labels
  6. Summary - Clustering

10 KMeans in depth

  1. KMeans algorithm
  2. Initialization
  3. n_init
  4. inertia_
  5. random_state
  6. Summary - KMeans

11 Hierarchical clustering

  1. What is hierarchical clustering?
  2. Dendrogram
  3. Agglomerative
  4. Linkage
  5. Advantages
  6. Summary - Hierarchical clustering

12 Project: Clustering

  1. Preparing data
  2. Training KMeans
  3. Label analysis
  4. Printing results

13 PCA

  1. What is PCA?
  2. Variance
  3. n_components
  4. fit_transform
  5. explained_variance_
  6. Summary - PCA

14 Dimensionality reduction

  1. What is dimensionality reduction?
  2. Why reduce?
  3. PCA vs t-SNE
  4. Visualization
  5. Advantages
  6. Summary - Dimensionality reduction

15 Feature selection

  1. What is feature selection?
  2. Variance threshold
  3. SelectKBest
  4. Recursive elimination
  5. Feature importance
  6. Summary - Feature selection

16 Imbalanced data

  1. What is imbalanced data?
  2. SMOTE
  3. Class weight
  4. Oversampling
  5. Metric selection
  6. Summary - Imbalanced data

17 Project: Features

  1. Feature extraction
  2. Feature selection
  3. Training model
  4. Evaluation

18 Ensemble methods

  1. What is ensemble?
  2. Voting
  3. Bagging
  4. Boosting
  5. Stacking
  6. Summary - Ensemble

19 Regularization

  1. What is regularization?
  2. L1 (Lasso)
  3. L2 (Ridge)
  4. alpha
  5. ElasticNet
  6. Summary - Regularization

20 Intermediate final

  1. Challenge 1
  2. Challenge 2
  3. Challenge 3
  4. Challenge 4