Machine Learning · Support vector machines

Many lines can separate two classes. An SVM picks the one with the largest margin, the widest street between the classes.

• Only the points on or inside the margin matter. These are the support vectors.
• C is the cost of margin violations. A small C gives a wider, softer margin (more regularised). A large C fits the training data more tightly.
• Kernel trick: replace dot products with a kernel function (RBF, polynomial). This gives non-linear boundaries without ever building the high-dimensional features. gamma sets how far one point’s influence reaches.

The make_moons data below can’t be separated by a straight line.

from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split
from sklearn.svm import SVC
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

X, y = make_moons(n_samples=400, noise=0.2, random_state=0)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, random_state=0)

for kernel, C in [("linear", 1), ("rbf", 0.1), ("rbf", 1), ("rbf", 100)]:
    m = make_pipeline(StandardScaler(), SVC(kernel=kernel, C=C, gamma="scale")).fit(X_tr, y_tr)
    svc = m[-1]
    print(f"{kernel:>6} C={C:<5}: test acc {m.score(X_te, y_te):.3f}, support vectors {svc.n_support_.sum()}")

Output:

linear C=1    : test acc 0.900, support vectors 97
   rbf C=0.1  : test acc 0.920, support vectors 147
   rbf C=1    : test acc 0.940, support vectors 76
   rbf C=100  : test acc 0.950, support vectors 30