Machine Learning
Why reduce?
Dimensionality reduction reduces computation cost and filters noise.
Why do we reduce?
The main reasons:
- Speed and memory: fewer columns, faster training.
- Noise reduction: low-variance components are often noise.
- Visualisation: down to 2-3 dimensions, the data can be plotted.
- The curse of dimensionality: in many dimensions points become almost equally far from each other; distance-based methods weaken.
rng = np.random.default_rng(0)
for d in [2, 100, 1000]:
A = rng.random((200, d))
m = np.linalg.norm(A[1:] - A[0], axis=1)
print(d, round(m.min() / m.max(), 2)) # as it nears 1, "nearest" loses meaning
Denoising
Reducing with few components and coming back projects the data onto the strongest patterns; noise in the low-variance components is discarded:
gurultulu = X + rng.normal(0, 4, X.shape)
pca = PCA(n_components=0.5).fit(gurultulu)
temiz = pca.inverse_transform(pca.transform(gurultulu))
print(((temiz - X) ** 2).mean() < ((gurultulu - X) ** 2).mean()) # True
If too few components are kept, real details are lost along with the noise; the balance is again found by validation.
Exercise Remove the noise
Add rng.normal(0, 4, X.shape) noise to the digit data. Fit a PCA keeping 50% of the variance to the noisy data and transform back. Print the mean squared error of the noisy and the denoised data against the original with round(..., 2) on one line.
Show example solution
import numpy as np
from sklearn.datasets import load_digits
X, y = load_digits(return_X_y=True)
from sklearn.decomposition import PCA
rng = np.random.default_rng(0)
gurultulu = X + rng.normal(0, 4, X.shape)
pca = PCA(n_components=0.5).fit(gurultulu)
temiz = pca.inverse_transform(pca.transform(gurultulu))
print(round(((gurultulu - X) ** 2).mean(), 2), round(((temiz - X) ** 2).mean(), 2))
Test yourself
-
In the curse of dimensionality, how does the ratio of nearest to farthest neighbour distances change?
- It approaches 0
- It doesn't change
- It becomes negative
- It approaches 1
Show answer
It approaches 1 — When all points are at similar distances, "nearest" neighbour information loses value.
-
Compute each row's distance to the first point.
m = np.linalg.norm(A[1:] - A[0], ____=1)
Show answer
m = np.linalg.norm(A[1:] - A[0], axis=1)
-
Match the reasons.
- Speed
- Fewer columns
- Noise
- Drop small components
- Visualization
- 2-3 dimensions
- Curse of dimensionality
- Distances even out
Next lesson: PCA vs t-SNE
