CodeBlanks Logo Icon CodeBlanks
Login Register

Feature selection

Feature selection simplifies the model by choosing the most informative features.

Why select features?

Not every feature is useful. Unnecessary features slow the model down, push it to learn noise and make interpretation harder. A model built with fewer but good features often generalises better.

from sklearn.feature_selection import VarianceThreshold
X = np.array([[1, 0, 3], [2, 0, 1], [3, 0, 2]])
print(VarianceThreshold(0.0).fit_transform(X))   # constant 2nd column dropped

The simplest selection is dropping columns that never change (variance 0).

Filtering by correlation

Features with no relationship to the target and features that are near-copies of each other are candidates for removal:

r = [abs(np.corrcoef(X[:, j], y)[0, 1]) for j in range(X.shape[1])]
print(np.round(r, 2))           # strength of relation to target
print(np.corrcoef(X, rowvar=False).round(2))   # between features

Correlation measures only linear relationships; at a later level you'll see model-based selection methods.

Practice Relation to the target

Compute the absolute correlation of each column with y as a list and print it with np.round(..., 2). Then print the index of the most strongly related column.

Starter code

model.py
import numpy as np

X = np.array([[1, 5, 2], [2, 3, 2], [3, 6, 2.1], [4, 2, 1.9], [5, 4, 2]])
y = np.array([2, 4, 6, 8, 10])

# buraya yaz

Sample solution

model.py
import numpy as np

X = np.array([[1, 5, 2], [2, 3, 2], [3, 6, 2.1], [4, 2, 1.9], [5, 4, 2]])
y = np.array([2, 4, 6, 8, 10])

r = [abs(np.corrcoef(X[:, j], y)[0, 1]) for j in range(X.shape[1])]
print(np.round(r, 2))
print(int(np.argmax(r)))

Expected output

[1.   0.3  0.22]
0

Questions and answers

Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.

1 What is the most likely harm of unnecessary features?

  • Better generalisation
  • Faster training
  • There is no harm
  • The model learning noise and overfitting Correct answer

Explanation: As the number of features grows, so does the chance of finding coincidental patterns.

2 Drop columns with variance below 0.1.

Question 
sec = VarianceThreshold(____=0.1)
Answer 
sec = VarianceThreshold(threshold=0.1)

Explanation: Features with variance below threshold are dropped.

3 Match the situations.

Term Meaning
Constant column Drop with variance threshold
Unrelated to the target Filter by correlation
Two duplicate columns Drop one
A few good features Better generalisation

All lessons in Beginner

1 What is ML?

  1. What is machine learning?
  2. Supervised and unsupervised
  3. Data and model
  4. Training and testing
  5. Feature and label notation
  6. Summary - ML fundamentals

2 NumPy basics

  1. What is NumPy?
  2. Creating arrays
  3. Array properties
  4. Array indexing
  5. Reshaping arrays
  6. Summary - NumPy basics

3 Pandas basics

  1. What is Pandas?
  2. Creating a DataFrame
  3. Column operations
  4. Selecting and filtering data
  5. Viewing first rows
  6. Summary - Pandas basics

4 Data preprocessing

  1. Data cleaning
  2. Missing data
  3. Data types
  4. Label encoding
  5. Data splitting
  6. Summary - Data preprocessing

5 Statistics basics

  1. Mean
  2. Median
  3. Standard deviation
  4. Variance
  5. Correlation
  6. Summary - Statistics

6 Linear algebra

  1. Vectors
  2. Matrices
  3. Matrix multiplication
  4. Transpose
  5. Identity matrix
  6. Summary - Linear algebra

7 Project: Data analysis

  1. Loading data
  2. Cleaning data
  3. Computing statistics
  4. Printing results

8 Regression

  1. What is regression?
  2. Linear regression
  3. Model training
  4. Prediction
  5. Error measurement
  6. Summary - Regression

9 KNN

  1. What is KNN?
  2. The K parameter
  3. Distance measurement
  4. Classification
  5. Model training
  6. Summary - KNN

10 Decision trees

  1. What is a decision tree?
  2. Splitting criterion
  3. Depth
  4. Classification
  5. Overfitting
  6. Summary - Decision trees

11 Classification

  1. What is classification?
  2. Binary classification
  3. Multi-class
  4. Accuracy
  5. Confusion matrix
  6. Summary - Classification

12 Model evaluation

  1. Train-test split
  2. Accuracy
  3. Precision
  4. Recall
  5. F1 score
  6. Summary - Model evaluation

13 Project: Classification

  1. Preparing data
  2. Training model
  3. Prediction
  4. Evaluation

14 Scaling

  1. What is scaling?
  2. StandardScaler
  3. MinMaxScaler
  4. When to use?
  5. Pipeline
  6. Summary - Scaling

15 Feature engineering

  1. What is a feature?
  2. Feature selection
  3. Feature extraction
  4. Categorical features
  5. Numerical features
  6. Summary - Feature engineering

16 Model selection

  1. What is model selection?
  2. Cross-validation
  3. Grid search
  4. Score comparison
  5. Hyperparameter
  6. Summary - Model selection

17 Project: Regression

  1. Preparing data
  2. Training model
  3. Prediction
  4. Error measurement

18 Metrics

  1. What is a metric?
  2. Accuracy
  3. Precision
  4. Recall
  5. F1 score
  6. Summary - Metrics

19 Data visualization

  1. What is visualization?
  2. Scatter plot
  3. Histogram
  4. Line plot
  5. Box plot
  6. Summary - Visualization

20 Beginner final

  1. Challenge 1
  2. Challenge 2
  3. Challenge 3
  4. Challenge 4