Machine Learning
PCA vs t-SNE
PCA is linear, t-SNE is a non-linear dimensionality reduction method.
PCA and t-SNE
t-SNE is a non-linear method that tries to keep points that are close in high dimensions close in 2 dimensions too. It preserves local neighbourhoods; that's why clusters separate very clearly visually:
from sklearn.manifold import TSNE
Z = TSNE(n_components=2, perplexity=30, random_state=0).fit_transform(X)
Differences: PCA is linear, fast and deterministic, and transform can be applied to new data. t-SNE is non-linear, slow, random and has no transform method.
Reading a t-SNE plot
Care is needed with a t-SNE plot:
- Cluster sizes are meaningless; dense clusters can expand.
- Distances between clusters aren't reliable.
perplexity changes the result a lot (5-50 is tried).- Different
random_state values give different layouts.
The only reliable thing is the local neighbourhood: points next to each other on the plot are close in the original space too.
Exercise Preservation of neighbourhoods
Reduce the first 500 examples of the digit data to 2 dimensions with PCA (2 components) and t-SNE (perplexity=30, random_state=0). Print the 5-fold CV accuracy of a 5-neighbour KNN on both 2-D representations with round(..., 3) on one line (PCA first). This measures how well local neighbourhoods are preserved.
Show example solution
import numpy as np
from sklearn.datasets import load_digits
X, y = load_digits(return_X_y=True)
from sklearn.decomposition import PCA
from sklearn.manifold import TSNE
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import cross_val_score
X, y = X[:500], y[:500]
Zp = PCA(n_components=2).fit_transform(X)
Zt = TSNE(n_components=2, perplexity=30, random_state=0).fit_transform(X)
knn = KNeighborsClassifier(n_neighbors=5)
print(round(cross_val_score(knn, Zp, y, cv=5).mean(), 3), round(cross_val_score(knn, Zt, y, cv=5).mean(), 3))
Test yourself
-
Why isn't t-SNE output used as input to a classifier?
- It always produces 3 dimensions
- It is used
- It can't be applied to new data and it distorts distances
- It is too accurate
Show answer
It can't be applied to new data and it distorts distances — t-SNE is a visualisation tool; there is no transform and distances between clusters aren't meaningful.
-
Set the neighbourhood scale for t-SNE.
Z = TSNE(n_components=2, ____=30, random_state=0).fit_transform(X)
Show answer
Z = TSNE(n_components=2, perplexity=30, random_state=0).fit_transform(X)
-
Match the properties.
- PCA
- Linear, fast
- t-SNE
- Non-linear, local
- transform
- Only in PCA
- perplexity
- t-SNE neighbourhood scale
Next lesson: Visualization
