PCA vs t-SNE
- Machine Learning course
- Intermediate
- Lesson 77/112
- 5 min
PCA is linear, t-SNE is a non-linear dimensionality reduction method.
PCA and t-SNE
t-SNE is a non-linear method that tries to keep points that are close in high dimensions close in 2 dimensions too. It preserves local neighbourhoods; that's why clusters separate very clearly visually:
from sklearn.manifold import TSNE
Z = TSNE(n_components=2, perplexity=30, random_state=0).fit_transform(X)Differences: PCA is linear, fast and deterministic, and transform can be applied to new data. t-SNE is non-linear, slow, random and has no transform method.
Reading a t-SNE plot
Care is needed with a t-SNE plot:
- Cluster sizes are meaningless; dense clusters can expand.
- Distances between clusters aren't reliable.
perplexitychanges the result a lot (5-50 is tried).- Different
random_statevalues give different layouts.
The only reliable thing is the local neighbourhood: points next to each other on the plot are close in the original space too.
Practice Preservation of neighbourhoods
Reduce the first 500 examples of the digit data to 2 dimensions with PCA (2 components) and t-SNE (perplexity=30, random_state=0). Print the 5-fold CV accuracy of a 5-neighbour KNN on both 2-D representations with round(..., 3) on one line (PCA first). This measures how well local neighbourhoods are preserved.
Starter code
import numpy as np
from sklearn.datasets import load_digits
X, y = load_digits(return_X_y=True)
from sklearn.decomposition import PCA
from sklearn.manifold import TSNE
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import cross_val_score
X, y = X[:500], y[:500]
# buraya yazSample solution
import numpy as np
from sklearn.datasets import load_digits
X, y = load_digits(return_X_y=True)
from sklearn.decomposition import PCA
from sklearn.manifold import TSNE
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import cross_val_score
X, y = X[:500], y[:500]
Zp = PCA(n_components=2).fit_transform(X)
Zt = TSNE(n_components=2, perplexity=30, random_state=0).fit_transform(X)
knn = KNeighborsClassifier(n_neighbors=5)
print(round(cross_val_score(knn, Zp, y, cv=5).mean(), 3), round(cross_val_score(knn, Zt, y, cv=5).mean(), 3))Expected output
0.682 0.952
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 Why isn't t-SNE output used as input to a classifier?
- It always produces 3 dimensions
- It is used
- It can't be applied to new data and it distorts distances Correct answer
- It is too accurate
Explanation: t-SNE is a visualisation tool; there is no transform and distances between clusters aren't meaningful.
2 Set the neighbourhood scale for t-SNE.
Z = TSNE(n_components=2, ____=30, random_state=0).fit_transform(X)Z = TSNE(n_components=2, perplexity=30, random_state=0).fit_transform(X)Explanation: perplexity is roughly the number of neighbours each point considers.
3 Match the properties.
| Term | Meaning |
|---|---|
PCA |
Linear, fast |
t-SNE |
Non-linear, local |
transform |
Only in PCA |
perplexity |
t-SNE neighbourhood scale |