ML Classification

Classification is the type of machine learning that predicts a category, such as "spam" or "not spam".

What is Classification?

In classification, the label y is a class (a category), not a number.

When there are two classes, it is binary classification. With more than two, it is multi-class classification. scikit-learn models handle both with the same code.

Logistic Regression

Despite its name, Logistic Regression is a classification model. It calculates the probability that an example belongs to a class, then picks the class with the highest probability.

Here we build a tiny SMS spam filter with two features: the number of links in a message, and the number of "spammy" words such as "free" or "winner".

from sklearn.linear_model import LogisticRegression

# [number of links, number of words like "free" or "winner"]
X = [[0, 0], [1, 0], [0, 1], [1, 1], [5, 4], [6, 3], [4, 5], [7, 6]]
y = [0, 0, 0, 0, 1, 1, 1, 1]          # 0 = normal, 1 = spam

model = LogisticRegression()
model.fit(X, y)

new_messages = [[0, 1], [6, 5]]
print(model.predict(new_messages))
print(model.predict_proba(new_messages).round(2))

[0 1]
[[0.96 0.04]
 [0.01 0.99]]

Reading the result

• predict() gives the class: the first message is normal (0), the second is spam (1).
• predict_proba() gives the probability of each class. The first message is 96% likely normal; the second is 99% likely spam.

A Multi-Class Example

The Iris dataset has three flower species. Logistic Regression handles all three:

from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(
    iris.data, iris.target, test_size=0.2, random_state=0, stratify=iris.target
)

model = LogisticRegression(max_iter=200)
model.fit(X_train, y_train)

flower = [[5.0, 3.4, 1.5, 0.2]]   # sepal length, sepal width, petal length, petal width (cm)
print(iris.target_names[model.predict(flower)])
print("Accuracy:", round(model.score(X_test, y_test), 2))

['setosa']
Accuracy: 1.0

score() returns accuracy for classifiers: the share of test examples predicted correctly. Here every test flower was right. Iris is an easy dataset; real problems rarely reach 100%.

max_iter=200 gives the model more steps to finish learning. If you see a "ConvergenceWarning", increase it or scale your features.

K-Nearest Neighbors (KNN)

KNN is the most intuitive classifier. To classify a new example, it finds the k closest examples in the training data and takes a vote. If 4 of the 5 nearest neighbours are spam, the new message is spam.

• Small k (like 1): follows the data very closely, can be fooled by noise
• Large k: smoother decisions, but may ignore small groups
• KNN measures distance, so scale your features first

Comparing Classifiers

Because all scikit-learn models share fit() and score(), trying several is easy:

from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=7, stratify=y
)

models = {
    "Logistic Regression": LogisticRegression(max_iter=200),
    "K-Nearest Neighbors": KNeighborsClassifier(n_neighbors=5),
    "Decision Tree": DecisionTreeClassifier(random_state=0),
}

for name, model in models.items():
    model.fit(X_train, y_train)
    print(name, round(model.score(X_test, y_test), 2))

Logistic Regression 0.93
K-Nearest Neighbors 0.96
Decision Tree 0.93

On this split KNN did slightly best, but with only 45 test flowers, the difference is small. A different random_state could change the order. Use cross-validation for fair comparisons.

Common Classification Models

Exercise