Machine Learning
Variance
PCA selects components that explain the most variance.
Preserving variance
PCA's goal is to keep as much of the total variance as possible while lowering the dimension. The variance each component carries is an eigenvalue of the data's covariance matrix; the component directions are the eigenvectors:
C = np.cov(Xs, rowvar=False)
ozdeger = np.linalg.eigvalsh(C)[::-1] # largest to smallest
print(np.allclose(ozdeger[:3], PCA(3).fit(Xs).explained_variance_)) # True
On scaled data each feature has variance 1, so the total variance equals the number of features.
The rotation preserves variance
If all components are kept, PCA is just a rotation: no information is lost and the total variance stays the same. Information is lost only when components are dropped:
tam = PCA().fit(Xs)
print(round(tam.explained_variance_.sum(), 3)) # ~13 (with the n-1 correction)
print(round(Xs.var(axis=0, ddof=1).sum(), 3)) # the same
PCA's variances are computed with ddof=1; the same correction must be used in the comparison.
Exercise Verify with eigenvalues
Sort the eigenvalues of the scaled wine data's covariance matrix from largest to smallest with np.linalg.eigvalsh. Print with np.allclose that the first three equal PCA(n_components=3).fit(Xs).explained_variance_. Then print the sum of all the eigenvalues with round(..., 3).
Show example solution
import numpy as np
from sklearn.datasets import load_wine
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
X, y = load_wine(return_X_y=True)
Xs = StandardScaler().fit_transform(X)
ozdeger = np.linalg.eigvalsh(np.cov(Xs, rowvar=False))[::-1]
print(np.allclose(ozdeger[:3], PCA(n_components=3).fit(Xs).explained_variance_))
print(round(ozdeger.sum(), 3))
Test yourself
-
For scaled data with 13 features (each variance ≈ 1), what is the total variance roughly?
Show answer
13 — The total variance is the sum of the column variances; the rotation preserves it.
-
Compute the covariance matrix between columns.
C = np.cov(Xs, ____=False)
Show answer
C = np.cov(Xs, rowvar=False)
-
Match the concepts.
- eigenvalue
- Component variance
- eigenvector
- Component direction
- all components
- Lossless rotation
- dropping components
- Information loss
Next lesson: n_components
