ML Evaluating Models

Evaluating a model means measuring how good its predictions are on data it has not seen.

Why One Number is Not Enough

Suppose 99 out of 100 bank transactions are normal and 1 is fraud. A lazy model that always says "normal" is 99% accurate, but it catches zero fraud. That is why we need more than accuracy.

Accuracy, Precision and Recall

Let's check a spam filter's predictions on 10 messages:

from sklearn.metrics import accuracy_score, confusion_matrix, precision_score, recall_score

# 1 = spam, 0 = normal
actual    = [1, 0, 1, 1, 0, 0, 1, 0, 0, 1]
predicted = [1, 0, 0, 1, 0, 1, 1, 0, 0, 0]

print("Accuracy: ", accuracy_score(actual, predicted))
print("Precision:", precision_score(actual, predicted))
print("Recall:   ", recall_score(actual, predicted))
print(confusion_matrix(actual, predicted))

Accuracy:  0.7
Precision: 0.75
Recall:    0.6
[[4 1]
 [2 3]]

The Confusion Matrix

The confusion matrix counts every kind of right and wrong answer. In scikit-learn, rows are the actual class and columns are the predicted class:

• True positive (TP): spam, correctly caught
• True negative (TN): normal, correctly let through
• False positive (FP): a normal message wrongly sent to spam
• False negative (FN): a spam message that got through

From these four counts:

The Classification Report

classification_report prints all the main metrics for each class at once. This example uses the breast cancer dataset that comes with scikit-learn:

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import classification_report

data = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(
    data.data, data.target, test_size=0.25, random_state=0, stratify=data.target
)

model = make_pipeline(StandardScaler(), LogisticRegression())
model.fit(X_train, y_train)

print(classification_report(y_test, model.predict(X_test), target_names=data.target_names))

              precision    recall  f1-score   support

   malignant       0.94      0.94      0.94        53
      benign       0.97      0.97      0.97        90

    accuracy                           0.96       143
   macro avg       0.96      0.96      0.96       143
weighted avg       0.96      0.96      0.96       143

support is the number of test examples in each class.

make_pipeline joins the scaler and the model into one object. When you call fit(), the scaler learns only from the training data, which avoids leaking test information.

Metrics for Regression

For models that predict numbers, we measure how far the predictions are from the real values:

from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score

actual_price    = [50000, 62000, 45000, 80000, 71000]   # Rs
predicted_price = [52000, 60000, 47000, 75000, 70000]

mae = mean_absolute_error(actual_price, predicted_price)
rmse = mean_squared_error(actual_price, predicted_price) ** 0.5
r2 = r2_score(actual_price, predicted_price)

print("MAE: ", round(mae))
print("RMSE:", round(rmse))
print("R2:  ", round(r2, 3))

MAE:  2400
RMSE: 2757
R2:   0.955

MAE and RMSE are in the same unit as the label, so they are easy to explain to a client: "Our price estimate is usually within about Rs 2,500."

Compare with a Baseline

A score means little on its own. Always compare with a simple baseline, such as always predicting the most common class or the average price. scikit-learn has DummyClassifier and DummyRegressor for this. If your model cannot beat the baseline, it has not learned anything useful.

Exercise