Machine Learning
Advantages
Dimensionality reduction reduces overfitting and increases model speed.
Advantages and cost
Dimensionality reduction speeds up a model and sometimes improves it; but it has a price: information loss and reduced interpretability. The way to measure the gain is comparing the same model with and without reduction:
for adim in ["passthrough", PCA(n_components=20)]:
pipe = make_pipeline(StandardScaler(), adim, SVC())
print(cross_val_score(pipe, X, y, cv=5).mean())
If success is similar and the time clearly shorter, the reduction pays off.
Non-linear alternatives
PCA misses non-linear structures. KernelPCA extends PCA to curved structures with the kernel trick and offers transform, so it can be used before a model:
from sklearn.decomposition import KernelPCA
kp = KernelPCA(n_components=2, kernel="rbf", gamma=10)
Z = kp.fit_transform(X_halka) # can separate nested rings
UMAP (a separate library) produces visuals similar to t-SNE but is faster and can be applied to new data.
Exercise Nested rings
Reduce the make_circles data to two components with PCA and with KernelPCA(kernel="rbf", gamma=10). Print the 5-fold CV accuracy of LogisticRegression() on each two-dimensional representation with round(..., 3) on one line (PCA first).
Show example solution
from sklearn.datasets import make_circles
from sklearn.decomposition import PCA, KernelPCA
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
X, y = make_circles(n_samples=400, factor=0.3, noise=0.05, random_state=0)
Zp = PCA(n_components=2).fit_transform(X)
Zk = KernelPCA(n_components=2, kernel="rbf", gamma=10).fit_transform(X)
m = LogisticRegression()
print(round(cross_val_score(m, Zp, y, cv=5).mean(), 3), round(cross_val_score(m, Zk, y, cv=5).mean(), 3))
Test yourself
-
What should be done if the CV score drops after reduction?
- Keep more components or remove the reduction
- Keep fewer components
- Look at the test set
- Nothing
Show answer
Keep more components or remove the reduction — The drop shows the dropped components carried important information.
-
Set up kernel PCA.
kp = ____(n_components=2, kernel="rbf", gamma=10)
Show answer
kp = KernelPCA(n_components=2, kernel="rbf", gamma=10)
-
Match the methods.
- PCA
- Linear, has transform
- KernelPCA
- Non-linear, has transform
- t-SNE
- Visualisation only
- passthrough
- Comparison without reduction
Next lesson: Summary - Dimensionality reduction
