The K parameter

K sets the number of neighbors; small K overfits, large K underfits.

Choosing K

If K is small the model fits every noisy example (overfitting); if K is large, distant examples vote too and the boundary flattens too much (underfitting).

for k in [1, 5, 15]:
    m = KNeighborsClassifier(n_neighbors=k).fit(X_train, y_train)
    print(k, m.score(X_train, y_train), m.score(X_test, y_test))

At K=1 training accuracy is always 1.0, because every training point's nearest neighbour is itself; this is misleading.

Odd numbers and validation

In two-class problems K is chosen as an odd number so the votes can't split evenly. The best K is chosen by looking at validation or cross-validation results, not the test set:

from sklearn.model_selection import cross_val_score
for k in [1, 3, 5, 7, 9]:
    s = cross_val_score(KNeighborsClassifier(k), X, y, cv=5).mean()
    print(k, round(s, 3))

You'll see cross-validation in detail at the intermediate level.

Exercise Compare K

Train a model for K = 1, 3, 5 and print K and the training accuracy on each line.

Show example solution

from sklearn.neighbors import KNeighborsClassifier

X = [[1], [2], [3], [4], [5], [6], [7], [8]]
y = [0, 0, 1, 0, 1, 1, 0, 1]

for k in [1, 3, 5]:
    m = KNeighborsClassifier(n_neighbors=k).fit(X, y)
    print(k, m.score(X, y))

Test yourself

Why is training accuracy usually 1.0 when K=1?
- Every training point's nearest neighbour is itself
- The model is perfect
- There is little data
- Distance can't be zero
Show answer

Every training point's nearest neighbour is itself — The model reads its own label; test accuracy is usually lower.

Take the neighbour count from the loop variable.

for k in [1, 3, 5]:
    m = KNeighborsClassifier(n_neighbors=____)

Show answer

for k in [1, 3, 5]:
    m = KNeighborsClassifier(n_neighbors=k)

Match the K values with the behaviour.

K = 1

Overly sensitive to noise
K too large

Boundary flattens too much
K is odd

Prevents tied votes
Cross-validation

Choosing K

Next lesson: Distance measurement

We Are Enhancing Your Experience

Cookie Preferences

Strictly Necessary Cookies

Analytical Cookies