CodeBlanks Logo Icon CodeBlanks
Login Register

Kernel

Kernel increases linear separability by mapping data to higher dimensions.

The kernel trick

Data that isn't linearly separable can become separable when moved to a higher-dimensional space. A kernel function computes the similarity between points directly, without doing that move explicitly:

SVC(kernel="linear")             # straight boundary
SVC(kernel="poly", degree=3)     # polynomial boundary
SVC(kernel="rbf", gamma=0.5)     # default; flexible, local boundaries

The RBF kernel exp(-γ‖x - x'‖²) measures how close two points are.

gamma

In RBF, gamma sets how narrow a point's area of influence is. Small gamma gives wide, smooth boundaries; large gamma draws small islands around each point (overfitting):

for g in [0.1, 1, 100]:
    m = SVC(gamma=g).fit(X_train, y_train)
    print(g, m.score(X_train, y_train), m.score(X_test, y_test))

The default gamma="scale" uses 1 / (n_features · X.var()) and is a good start on scaled data.

Practice Compare kernels

Train SVCs with the linear, poly and rbf kernels on the moon data. On each line print the kernel name and the test accuracy with round(..., 3).

Starter code

model.py
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split
from sklearn.svm import SVC

X, y = make_moons(n_samples=300, noise=0.25, random_state=0)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)

# buraya yaz

Sample solution

model.py
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split
from sklearn.svm import SVC

X, y = make_moons(n_samples=300, noise=0.25, random_state=0)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)

for k in ["linear", "poly", "rbf"]:
    m = SVC(kernel=k).fit(X_train, y_train)
    print(k, round(m.score(X_test, y_test), 3))

Expected output

linear 0.856
poly 0.811
rbf 0.967

Questions and answers

Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.

1 Which kernel suits two moon-shaped classes (make_moons) best?

  • linear
  • The kernel doesn't matter
  • None can learn it
  • rbf Correct answer

Explanation: A curved boundary can't be drawn with the linear kernel; rbf gives local flexibility.

2 Use the polynomial kernel.

Question 
m = SVC(kernel=____, degree=3)
Answer 
m = SVC(kernel="poly", degree=3)

Explanation: degree is used only with the poly kernel.

3 Match the kernels.

Term Meaning
linear Straight boundary
poly Polynomial curve
rbf Local similarity
large gamma Islands, overfitting

All lessons in Intermediate

1 scikit-learn basics

  1. What is scikit-learn?
  2. Estimator API
  3. The fit method
  4. The predict method
  5. The score method
  6. Summary - scikit-learn

2 Pipeline

  1. What is a Pipeline?
  2. Steps
  3. fit and predict
  4. Advantages
  5. ColumnTransformer
  6. Summary - Pipeline

3 Cross-validation

  1. What is cross-validation?
  2. K-fold
  3. cross_val_score
  4. Stratified
  5. Advantages
  6. Summary - Cross-validation

4 Grid search

  1. What is grid search?
  2. GridSearchCV
  3. param_grid
  4. best_params_
  5. best_score_
  6. Summary - Grid search

5 SVM

  1. What is SVM?
  2. Margin
  3. Kernel
  4. C parameter
  5. Classification
  6. Summary - SVM

6 Project: SVM

  1. Preparing data
  2. Training model
  3. Prediction
  4. Evaluation

7 Random forest

  1. What is random forest?
  2. Ensemble
  3. n_estimators
  4. Bagging
  5. Feature importance
  6. Summary - Random forest

8 Boosting

  1. What is boosting?
  2. AdaBoost
  3. Gradient boosting
  4. Learning rate
  5. Advantages
  6. Summary - Boosting

9 Clustering

  1. What is clustering?
  2. KMeans
  3. Centroid
  4. Elbow method
  5. Labels
  6. Summary - Clustering

10 KMeans in depth

  1. KMeans algorithm
  2. Initialization
  3. n_init
  4. inertia_
  5. random_state
  6. Summary - KMeans

11 Hierarchical clustering

  1. What is hierarchical clustering?
  2. Dendrogram
  3. Agglomerative
  4. Linkage
  5. Advantages
  6. Summary - Hierarchical clustering

12 Project: Clustering

  1. Preparing data
  2. Training KMeans
  3. Label analysis
  4. Printing results

13 PCA

  1. What is PCA?
  2. Variance
  3. n_components
  4. fit_transform
  5. explained_variance_
  6. Summary - PCA

14 Dimensionality reduction

  1. What is dimensionality reduction?
  2. Why reduce?
  3. PCA vs t-SNE
  4. Visualization
  5. Advantages
  6. Summary - Dimensionality reduction

15 Feature selection

  1. What is feature selection?
  2. Variance threshold
  3. SelectKBest
  4. Recursive elimination
  5. Feature importance
  6. Summary - Feature selection

16 Imbalanced data

  1. What is imbalanced data?
  2. SMOTE
  3. Class weight
  4. Oversampling
  5. Metric selection
  6. Summary - Imbalanced data

17 Project: Features

  1. Feature extraction
  2. Feature selection
  3. Training model
  4. Evaluation

18 Ensemble methods

  1. What is ensemble?
  2. Voting
  3. Bagging
  4. Boosting
  5. Stacking
  6. Summary - Ensemble

19 Regularization

  1. What is regularization?
  2. L1 (Lasso)
  3. L2 (Ridge)
  4. alpha
  5. ElasticNet
  6. Summary - Regularization

20 Intermediate final

  1. Challenge 1
  2. Challenge 2
  3. Challenge 3
  4. Challenge 4