Machine Learning
C parameter
C controls the trade-off between margin width and classification error.
The C parameter
C is the penalty for margin violations. A large C counts training errors as very costly: the model tries to classify every point correctly and the boundary bends. A small C accepts some errors and the boundary is smoother:
for C in [0.01, 1, 100]:
m = SVC(C=C).fit(X_train, y_train)
print(C, len(m.support_), m.score(X_test, y_test))
With a small C many more points become support vectors, because every point inside the wide margin affects the boundary.
C and gamma together
In an RBF SVM, C and gamma set the complexity together; neither can be tuned alone. The typical approach is to search both together on a logarithmic grid:
izgara = {"svc__C": np.logspace(-2, 3, 6), "svc__gamma": np.logspace(-3, 1, 5)}
gs = GridSearchCV(make_pipeline(StandardScaler(), SVC()), izgara, cv=5)
The combination of high C with high gamma is the most flexible, most overfitting model.
Exercise The effect of C
Train an RBF SVC on the moon data for C = 0.01, 1, 100. On each line print C, the number of support vectors (len(m.support_)) and the test accuracy with round(..., 3).
Show example solution
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split
from sklearn.svm import SVC
X, y = make_moons(n_samples=300, noise=0.25, random_state=0)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)
for C in [0.01, 1, 100]:
m = SVC(C=C).fit(X_train, y_train)
print(C, len(m.support_), round(m.score(X_test, y_test), 3))
Test yourself
-
How does the number of support vectors usually change as C grows?
- It doesn't change
- It becomes zero
- It decreases
- It increases
Show answer
It decreases — As the margin narrows, fewer points stay inside it.
-
Get the indices of the support vectors.
idx = m.____
Show answer
idx = m.support_
-
Match the settings with the behaviour.
- small C
- Wide margin, many support vectors
- large C
- Narrow margin, tight fit
- small gamma
- Smooth boundary
- large C and gamma
- Most overfitting
Next lesson: Classification
