The K parameter
- Machine Learning course
- Beginner
- Lesson 48/112
- 5 min
K sets the number of neighbors; small K overfits, large K underfits.
Choosing K
If K is small the model fits every noisy example (overfitting); if K is large, distant examples vote too and the boundary flattens too much (underfitting).
for k in [1, 5, 15]:
m = KNeighborsClassifier(n_neighbors=k).fit(X_train, y_train)
print(k, m.score(X_train, y_train), m.score(X_test, y_test))At K=1 training accuracy is always 1.0, because every training point's nearest neighbour is itself; this is misleading.
Odd numbers and validation
In two-class problems K is chosen as an odd number so the votes can't split evenly. The best K is chosen by looking at validation or cross-validation results, not the test set:
from sklearn.model_selection import cross_val_score
for k in [1, 3, 5, 7, 9]:
s = cross_val_score(KNeighborsClassifier(k), X, y, cv=5).mean()
print(k, round(s, 3))You'll see cross-validation in detail at the intermediate level.
Practice Compare K
Train a model for K = 1, 3, 5 and print K and the training accuracy on each line.
Starter code
from sklearn.neighbors import KNeighborsClassifier
X = [[1], [2], [3], [4], [5], [6], [7], [8]]
y = [0, 0, 1, 0, 1, 1, 0, 1]
# buraya yazSample solution
from sklearn.neighbors import KNeighborsClassifier
X = [[1], [2], [3], [4], [5], [6], [7], [8]]
y = [0, 0, 1, 0, 1, 1, 0, 1]
for k in [1, 3, 5]:
m = KNeighborsClassifier(n_neighbors=k).fit(X, y)
print(k, m.score(X, y))Expected output
1 1.0 3 0.625 5 0.625
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 Why is training accuracy usually 1.0 when K=1?
- Every training point's nearest neighbour is itself Correct answer
- The model is perfect
- There is little data
- Distance can't be zero
Explanation: The model reads its own label; test accuracy is usually lower.
2 Take the neighbour count from the loop variable.
for k in [1, 3, 5]:
m = KNeighborsClassifier(n_neighbors=____)for k in [1, 3, 5]:
m = KNeighborsClassifier(n_neighbors=k)Explanation: A separate model is built for each k in the loop.
3 Match the K values with the behaviour.
| Term | Meaning |
|---|---|
K = 1 |
Overly sensitive to noise |
K too large |
Boundary flattens too much |
K is odd |
Prevents tied votes |
Cross-validation |
Choosing K |