Why reduce?
- Machine Learning course
- Intermediate
- Lesson 76/112
- 5 min
Dimensionality reduction reduces computation cost and filters noise.
Why do we reduce?
The main reasons:
- Speed and memory: fewer columns, faster training.
- Noise reduction: low-variance components are often noise.
- Visualisation: down to 2-3 dimensions, the data can be plotted.
- The curse of dimensionality: in many dimensions points become almost equally far from each other; distance-based methods weaken.
rng = np.random.default_rng(0)
for d in [2, 100, 1000]:
A = rng.random((200, d))
m = np.linalg.norm(A[1:] - A[0], axis=1)
print(d, round(m.min() / m.max(), 2)) # as it nears 1, "nearest" loses meaningDenoising
Reducing with few components and coming back projects the data onto the strongest patterns; noise in the low-variance components is discarded:
gurultulu = X + rng.normal(0, 4, X.shape)
pca = PCA(n_components=0.5).fit(gurultulu)
temiz = pca.inverse_transform(pca.transform(gurultulu))
print(((temiz - X) ** 2).mean() < ((gurultulu - X) ** 2).mean()) # TrueIf too few components are kept, real details are lost along with the noise; the balance is again found by validation.
Practice Remove the noise
Add rng.normal(0, 4, X.shape) noise to the digit data. Fit a PCA keeping 50% of the variance to the noisy data and transform back. Print the mean squared error of the noisy and the denoised data against the original with round(..., 2) on one line.
Starter code
import numpy as np
from sklearn.datasets import load_digits
X, y = load_digits(return_X_y=True)
from sklearn.decomposition import PCA
rng = np.random.default_rng(0)
# buraya yazSample solution
import numpy as np
from sklearn.datasets import load_digits
X, y = load_digits(return_X_y=True)
from sklearn.decomposition import PCA
rng = np.random.default_rng(0)
gurultulu = X + rng.normal(0, 4, X.shape)
pca = PCA(n_components=0.5).fit(gurultulu)
temiz = pca.inverse_transform(pca.transform(gurultulu))
print(round(((gurultulu - X) ** 2).mean(), 2), round(((temiz - X) ** 2).mean(), 2))Expected output
16.01 7.22
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 In the curse of dimensionality, how does the ratio of nearest to farthest neighbour distances change?
- It approaches 0
- It doesn't change
- It becomes negative
- It approaches 1 Correct answer
Explanation: When all points are at similar distances, "nearest" neighbour information loses value.
2 Compute each row's distance to the first point.
m = np.linalg.norm(A[1:] - A[0], ____=1)m = np.linalg.norm(A[1:] - A[0], axis=1)Explanation: axis=1 gives a separate norm for each row.
3 Match the reasons.
| Term | Meaning |
|---|---|
Speed |
Fewer columns |
Noise |
Drop small components |
Visualization |
2-3 dimensions |
Curse of dimensionality |
Distances even out |