Machine Learning · Cross-validation

One random train/test split gives a noisy estimate: another split could give a quite different number. k-fold CV splits the data into k folds, trains on k − 1 of them, tests on the remaining one, and rotates. You get k scores, so you can report a mean ± spread.

• StratifiedKFold keeps class proportions the same in each fold (it’s the default for classifiers).
• GroupKFold keeps all rows from one group (a patient, a user) inside a single fold. Without it you get leakage through near-duplicates.
• Leakage: any preprocessing fitted on all the data (scaling, feature selection) sees the test folds. The fix is to put it inside a Pipeline, so it’s refitted inside each fold.

The demo shows feature selection leakage. The data is pure noise, so honest accuracy should be about 50%.

import numpy as np
from sklearn.model_selection import train_test_split, cross_val_score, StratifiedKFold
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline

rng = np.random.default_rng(0)
X = rng.normal(size=(100, 5000))          # pure noise
y = rng.integers(0, 2, 100)

scores = [LogisticRegression(max_iter=1000).fit(*train_test_split(X, y, random_state=s)[0::2]).score(*train_test_split(X, y, random_state=s)[1::2]) for s in range(5)]
print("single splits vary:", np.round(scores, 2))

cv = StratifiedKFold(5, shuffle=True, random_state=0)
X_leaky = SelectKBest(f_classif, k=20).fit_transform(X, y)      # selection saw all labels!
leaky = cross_val_score(LogisticRegression(max_iter=1000), X_leaky, y, cv=cv)
honest = cross_val_score(make_pipeline(SelectKBest(f_classif, k=20), LogisticRegression(max_iter=1000)), X, y, cv=cv)
print(f"leaky CV accuracy : {leaky.mean():.2f} ± {leaky.std():.2f}")
print(f"honest CV accuracy: {honest.mean():.2f} ± {honest.std():.2f}")

Output:

single splits vary: [0.4  0.48 0.28 0.36 0.52]
leaky CV accuracy : 0.87 ± 0.05
honest CV accuracy: 0.50 ± 0.09