Machine Learning · Model evaluation metrics

Classification, starting from the confusion matrix (TP, FP, FN, TN):

metric	formula	answers
accuracy	(TP+TN)/all	how often am I right? (misleading when classes are imbalanced)
precision	TP/(TP+FP)	of the cases I flagged, how many were real?
recall	TP/(TP+FN)	of the real cases, how many did I catch?
F1	harmonic mean of P and R	one balanced number
ROC-AUC	area under the TPR–FPR curve	ranking quality across all thresholds

Regression: MAE, RMSE (punishes big errors), R².

The demo is a heavily imbalanced problem (about 1.5% positives). A “model” that always says “negative” scores about 98.5% accuracy and catches nothing.

import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, confusion_matrix, classification_report, roc_auc_score

X, y = make_classification(n_samples=20_000, weights=[0.99], n_informative=5, random_state=0)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, stratify=y, random_state=0)
print("positive rate:", y.mean().round(4))

dummy = DummyClassifier(strategy="most_frequent").fit(X_tr, y_tr)
print("dummy accuracy:", round(accuracy_score(y_te, dummy.predict(X_te)), 4), "<- catches zero positives")

clf = LogisticRegression(max_iter=1000, class_weight="balanced").fit(X_tr, y_tr)
pred = clf.predict(X_te)
print(confusion_matrix(y_te, pred))
print(classification_report(y_te, pred, digits=3))
print("ROC-AUC:", round(roc_auc_score(y_te, clf.predict_proba(X_te)[:, 1]), 3))

Output:

positive rate: 0.015
dummy accuracy: 0.985 <- catches zero positives
[[4126  799]
 [  34   41]]
              precision    recall  f1-score   support

           0      0.992     0.838     0.908      4925
           1      0.049     0.547     0.090        75

    accuracy                          0.833      5000
   macro avg      0.520     0.692     0.499      5000
weighted avg      0.978     0.833     0.896      5000

ROC-AUC: 0.752

The real model catches more than half the positives, but most of the cases it flags are false alarms (low precision). Accuracy alone would have hidden both facts.