Machine Learning · k-nearest neighbours

k-NN is the simplest non-parametric model. To predict, find the k closest training points and take a majority vote (for regression, the average).

• There’s no real training step. All the work happens at prediction time, so it’s slow on big data.
• Distance is everything. If one feature is measured in thousands and another in fractions, the big one dominates. Always scale first.
• k controls flexibility. k = 1 memorises the training set (high variance). A large k smooths everything out (high bias).

I swept k on the wine dataset, with and without scaling.

from sklearn.datasets import load_wine
from sklearn.model_selection import cross_val_score
from sklearn.neighbors import KNeighborsClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

X, y = load_wine(return_X_y=True)
print("feature ranges differ a lot:", X.min(0).round(1)[:4], "...", X.max(0).round(1)[-2:])

for k in [1, 5, 15, 51]:
    raw = cross_val_score(KNeighborsClassifier(k), X, y, cv=5).mean()
    scaled = cross_val_score(make_pipeline(StandardScaler(), KNeighborsClassifier(k)), X, y, cv=5).mean()
    print(f"k={k:>2}: unscaled {raw:.3f} | scaled {scaled:.3f}")

Output:

feature ranges differ a lot: [11.   0.7  1.4 10.6] ... [   4. 1680.]
k= 1: unscaled 0.725 | scaled 0.950
k= 5: unscaled 0.691 | scaled 0.949
k=15: unscaled 0.714 | scaled 0.955
k=51: unscaled 0.714 | scaled 0.961