Variance
- Machine Learning course
- Intermediate
- Lesson 70/112
- 5 min
PCA selects components that explain the most variance.
Preserving variance
PCA's goal is to keep as much of the total variance as possible while lowering the dimension. The variance each component carries is an eigenvalue of the data's covariance matrix; the component directions are the eigenvectors:
C = np.cov(Xs, rowvar=False)
ozdeger = np.linalg.eigvalsh(C)[::-1] # largest to smallest
print(np.allclose(ozdeger[:3], PCA(3).fit(Xs).explained_variance_)) # TrueOn scaled data each feature has variance 1, so the total variance equals the number of features.
The rotation preserves variance
If all components are kept, PCA is just a rotation: no information is lost and the total variance stays the same. Information is lost only when components are dropped:
tam = PCA().fit(Xs)
print(round(tam.explained_variance_.sum(), 3)) # ~13 (with the n-1 correction)
print(round(Xs.var(axis=0, ddof=1).sum(), 3)) # the samePCA's variances are computed with ddof=1; the same correction must be used in the comparison.
Practice Verify with eigenvalues
Sort the eigenvalues of the scaled wine data's covariance matrix from largest to smallest with np.linalg.eigvalsh. Print with np.allclose that the first three equal PCA(n_components=3).fit(Xs).explained_variance_. Then print the sum of all the eigenvalues with round(..., 3).
Starter code
import numpy as np
from sklearn.datasets import load_wine
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
X, y = load_wine(return_X_y=True)
Xs = StandardScaler().fit_transform(X)
# buraya yazSample solution
import numpy as np
from sklearn.datasets import load_wine
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
X, y = load_wine(return_X_y=True)
Xs = StandardScaler().fit_transform(X)
ozdeger = np.linalg.eigvalsh(np.cov(Xs, rowvar=False))[::-1]
print(np.allclose(ozdeger[:3], PCA(n_components=3).fit(Xs).explained_variance_))
print(round(ozdeger.sum(), 3))Expected output
True 13.073
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 For scaled data with 13 features (each variance ≈ 1), what is the total variance roughly?
- 0
- 13 Correct answer
- 1
- 169
Explanation: The total variance is the sum of the column variances; the rotation preserves it.
2 Compute the covariance matrix between columns.
C = np.cov(Xs, ____=False)C = np.cov(Xs, rowvar=False)Explanation: rowvar=False treats each column as a variable.
3 Match the concepts.
| Term | Meaning |
|---|---|
eigenvalue |
Component variance |
eigenvector |
Component direction |
all components |
Lossless rotation |
dropping components |
Information loss |