Summary - KNN

In this group you learned the KNN algorithm, the K parameter, and classification.

Summary: KNN

In this group:

KNN classifies by the majority vote of the K nearest examples; fit only stores the data.
Small K overfits, large K underfits; K is chosen by validation, odd numbers preferred for two classes.
Euclidean (default) and Manhattan distances; the metric parameter.
kneighbors, predict_proba, weights="distance".
Because it is distance-based, scaling is a must.

For regression the same idea lives in KNeighborsRegressor: the neighbours' mean is taken.

Exercise The best K

For K = 1, 3, 5, 7, 9 compute the mean 5-fold cross-validation score on Iris (cross_val_score(..., cv=5).mean()). Print the K that gives the best score.

Show example solution

from sklearn.datasets import load_iris
from sklearn.model_selection import cross_val_score
from sklearn.neighbors import KNeighborsClassifier

X, y = load_iris(return_X_y=True)

skorlar = {}
for k in [1, 3, 5, 7, 9]:
    skorlar[k] = cross_val_score(KNeighborsClassifier(n_neighbors=k), X, y, cv=5).mean()
print(max(skorlar, key=skorlar.get))

Test yourself

Which is the most important preprocessing step when working with KNN?
- Scaling the features
- Shuffling the labels
- Sorting the columns
- None is needed
Show answer

Scaling the features — Distances are computed from the differences of all features; units must be equalised.

Weight the neighbours by distance.

m = KNeighborsClassifier(n_neighbors=5, weights=____)

Show answer

m = KNeighborsClassifier(n_neighbors=5, weights="distance")

Match the parameters.

n_neighbors

The K value
metric

Distance type
weights

Vote weighting
KNeighborsRegressor

Neighbour mean

Next lesson: What is a decision tree?

We Are Enhancing Your Experience

Cookie Preferences

Strictly Necessary Cookies

Analytical Cookies