Advantages
- Machine Learning course
- Intermediate
- Lesson 79/112
- 5 min
Dimensionality reduction reduces overfitting and increases model speed.
Advantages and cost
Dimensionality reduction speeds up a model and sometimes improves it; but it has a price: information loss and reduced interpretability. The way to measure the gain is comparing the same model with and without reduction:
for adim in ["passthrough", PCA(n_components=20)]:
pipe = make_pipeline(StandardScaler(), adim, SVC())
print(cross_val_score(pipe, X, y, cv=5).mean())If success is similar and the time clearly shorter, the reduction pays off.
Non-linear alternatives
PCA misses non-linear structures. KernelPCA extends PCA to curved structures with the kernel trick and offers transform, so it can be used before a model:
from sklearn.decomposition import KernelPCA
kp = KernelPCA(n_components=2, kernel="rbf", gamma=10)
Z = kp.fit_transform(X_halka) # can separate nested ringsUMAP (a separate library) produces visuals similar to t-SNE but is faster and can be applied to new data.
Practice Nested rings
Reduce the make_circles data to two components with PCA and with KernelPCA(kernel="rbf", gamma=10). Print the 5-fold CV accuracy of LogisticRegression() on each two-dimensional representation with round(..., 3) on one line (PCA first).
Starter code
from sklearn.datasets import make_circles
from sklearn.decomposition import PCA, KernelPCA
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
X, y = make_circles(n_samples=400, factor=0.3, noise=0.05, random_state=0)
# buraya yazSample solution
from sklearn.datasets import make_circles
from sklearn.decomposition import PCA, KernelPCA
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
X, y = make_circles(n_samples=400, factor=0.3, noise=0.05, random_state=0)
Zp = PCA(n_components=2).fit_transform(X)
Zk = KernelPCA(n_components=2, kernel="rbf", gamma=10).fit_transform(X)
m = LogisticRegression()
print(round(cross_val_score(m, Zp, y, cv=5).mean(), 3), round(cross_val_score(m, Zk, y, cv=5).mean(), 3))Expected output
0.435 1.0
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 What should be done if the CV score drops after reduction?
- Keep more components or remove the reduction Correct answer
- Keep fewer components
- Look at the test set
- Nothing
Explanation: The drop shows the dropped components carried important information.
2 Set up kernel PCA.
kp = ____(n_components=2, kernel="rbf", gamma=10)kp = KernelPCA(n_components=2, kernel="rbf", gamma=10)Explanation: KernelPCA lives in sklearn.decomposition.
3 Match the methods.
| Term | Meaning |
|---|---|
PCA |
Linear, has transform |
KernelPCA |
Non-linear, has transform |
t-SNE |
Visualisation only |
passthrough |
Comparison without reduction |