Machine Learning
What is PCA?
PCA reduces dimensions while preserving the most variance in data.
What is PCA?
Principal component analysis (PCA) is a linear transformation that rotates the data towards the directions in which it spreads most. The first component is the direction of largest variance, the second is perpendicular to it with the largest remaining variance, and so on.
from sklearn.decomposition import PCA
pca = PCA(n_components=2)
Z = pca.fit_transform(Xs) # 13 features -> 2 components
print(Z.shape)
If a few components carry most of the data's information, the dimension can be reduced a lot.
Scale first
Since PCA looks at variance, the feature with large units takes over the first component. On the wine data, if PCA is done without scaling, the first component is almost entirely proline (~1000):
ham = PCA(n_components=1).fit(X)
print(abs(ham.components_[0]).argmax()) # 12: proline
olcekli = PCA(n_components=1).fit(Xs)
print(abs(olcekli.components_[0]).round(2)) # the load spreads over many features
With features in different units, StandardScaler before PCA is almost always necessary.
Exercise The effect of scale
Train a one-component PCA on the raw X and on the scaled Xs. For each, print the index of the feature with the largest absolute loading in the first component on one line (raw first).
Show example solution
import numpy as np
from sklearn.datasets import load_wine
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
X, y = load_wine(return_X_y=True)
Xs = StandardScaler().fit_transform(X)
ham = PCA(n_components=1).fit(X)
olcekli = PCA(n_components=1).fit(Xs)
print(abs(ham.components_[0]).argmax(), abs(olcekli.components_[0]).argmax())
Test yourself
-
How is PCA's second component related to the first?
- It is the same direction
- No relation, it is random
- It is perpendicular (uncorrelated)
- It is parallel to it
Show answer
It is perpendicular (uncorrelated) — The components are perpendicular; there is no correlation between the transformed columns.
-
Reduce the data to two principal components.
Z = PCA(____=2).fit_transform(Xs)
Show answer
Z = PCA(n_components=2).fit_transform(Xs)
-
Match the concepts.
- 1st component
- Direction of largest variance
- 2nd component
- Perpendicular to the first
- components_
- Component directions
- scaling
- Prevents unit dominance
Next lesson: Variance
