Machine Learning
Margin
The margin is the distance from the hyperplane to the nearest points of each class.
The margin
The margin is the gap between the boundary and the nearest points. A linear SVM chooses the boundary w·x + b = 0 so that the margin width 2 / ‖w‖ is as large as possible. A wide margin means safer decisions on new points.
m = SVC(kernel="linear", C=1000).fit(X, y) # close to a hard margin
w = m.coef_[0]
print(2 / np.linalg.norm(w)) # margin width
For the support vectors |w·x + b| = 1; these points sit on the edge of the margin.
Hard and soft margin
A hard margin allows no point inside the margin; if the data can't be separated perfectly there is no solution, and a single outlier shifts the boundary a lot. A soft margin allows some violations at a cost; the balance is set with C.
# large C: violations costly -> narrow margin, tight fit to training
# small C: violations cheap -> wide margin, smoother boundary
Almost all real data is noisy, so in practice a soft margin is always used.
Exercise Measure the margin
Train an SVC(kernel="linear", C=1000). Compute the margin width with 2 / np.linalg.norm(m.coef_[0]) and print it with round(..., 3). Then print the absolute decision_function values at the support vectors with np.round(..., 2).
Show example solution
import numpy as np
from sklearn.svm import SVC
X = np.array([[0, 0], [1, 0], [0, 1], [3, 3], [4, 3], [3, 4]])
y = np.array([0, 0, 0, 1, 1, 1])
m = SVC(kernel="linear", C=1000).fit(X, y)
print(round(2 / np.linalg.norm(m.coef_[0]), 3))
print(np.round(np.abs(m.decision_function(m.support_vectors_)), 2))
Test yourself
-
What does widening the margin mean for ‖w‖?
- ‖w‖ shrinks
- ‖w‖ grows
- ‖w‖ doesn't change
- ‖w‖ becomes zero
Show answer
‖w‖ shrinks — The margin is 2/‖w‖, so as the weight norm shrinks the margin widens.
-
Compute the norm of the weight vector.
marj = 2 / np.linalg.____(m.coef_[0])
Show answer
marj = 2 / np.linalg.norm(m.coef_[0])
-
Match the concepts.
- margin
- 2 / ‖w‖
- hard margin
- No violations allowed
- soft margin
- Penalised violations
- |w·x + b| = 1
- Support vectors
Next lesson: Kernel
