Machine Learning · PCA

PCA finds a new set of orthogonal axes:

• the 1st principal component is the direction of maximum variance
• the 2nd is the next-largest variance, orthogonal to the first, and so on

Projecting onto the first few components gives dimensionality reduction while keeping most of the information. Mathematically, the components are eigenvectors of the covariance matrix (or right singular vectors of the centred data). That’s the eigh from the NumPy linear algebra post.

explained_variance_ratio_ tells you how much each component keeps. The digits dataset has 64 pixels per image. I checked how many components it takes to keep 90% of the variance, and whether a classifier still works afterwards.

import numpy as np
from sklearn.datasets import load_digits
from sklearn.decomposition import PCA
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_digits(return_X_y=True)
pca = PCA().fit(X)
cum = np.cumsum(pca.explained_variance_ratio_)
k90 = int(np.searchsorted(cum, 0.90) + 1)
print("dims:", X.shape[1], "| first 3 ratios:", pca.explained_variance_ratio_[:3].round(3), "| components for 90%:", k90)

# same result as eigen-decomposing the covariance matrix
vals = np.linalg.eigvalsh(np.cov(X, rowvar=False))[::-1]
print("matches covariance eigenvalues:", np.allclose(vals[:5], pca.explained_variance_[:5]))

for n in [64, k90, 10]:
    m = make_pipeline(StandardScaler(), PCA(n), LogisticRegression(max_iter=3000))
    print(f"{n:>2} components -> CV accuracy {cross_val_score(m, X, y, cv=5).mean():.3f}")

Output:

dims: 64 | first 3 ratios: [0.149 0.136 0.118] | components for 90%: 21
matches covariance eigenvalues: True
64 components -> CV accuracy 0.920
21 components -> CV accuracy 0.899
10 components -> CV accuracy 0.840