ML Overfitting and Underfitting

Overfitting happens when a model learns the training data too well, including its noise, and then fails on new data.

Overfitting and Underfitting

Think of two students preparing for an exam:

• Ali memorises the answers to last year's paper word for word. He scores 100% on that paper but fails when the questions change. This is overfitting.
• Bilal barely studies and only knows a few general ideas. He does badly on every paper. This is underfitting.
• Ayesha understands the concepts. She does well on old papers and new ones. This is a good fit.

Seeing Overfitting in Code

The best way to spot overfitting is to compare training and test scores. Here we train decision trees of different depths:

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)

for depth in [1, 3, None]:
    tree = DecisionTreeClassifier(max_depth=depth, random_state=0)
    tree.fit(X_train, y_train)
    print(f"max_depth={depth}: train {tree.score(X_train, y_train):.2f}, test {tree.score(X_test, y_test):.2f}")

max_depth=1: train 0.93, test 0.89
max_depth=3: train 0.97, test 0.95
max_depth=None: train 1.00, test 0.91

Reading the result

• Depth 1 is too simple: both scores are lower. Slight underfitting.
• Depth 3 is the sweet spot: high scores, small gap.
• No limit gets a perfect 1.00 on training data but drops on the test data. It memorised the training set. Classic overfitting.

Cross-Validation

One train/test split can be lucky or unlucky. Cross-validation gives a more reliable score. With 5-fold cross-validation, the data is cut into 5 parts. The model trains on 4 parts and tests on the 5th, five times, so every part is used for testing once.

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score
from sklearn.tree import DecisionTreeClassifier

X, y = load_breast_cancer(return_X_y=True)

tree = DecisionTreeClassifier(max_depth=3, random_state=0)
scores = cross_val_score(tree, X, y, cv=5)

print("Scores:", scores.round(2))
print("Average:", round(scores.mean(), 2))

Scores: [0.91 0.89 0.94 0.95 0.89]
Average: 0.92

The average (0.92) is a fairer estimate than any single split. The spread (0.89 to 0.95) shows how much the score can change by chance.

Tuning with GridSearchCV

Settings like max_depth are called hyperparameters: you choose them, the model does not learn them. GridSearchCV tries every combination you list and uses cross-validation to pick the best:

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import GridSearchCV
from sklearn.tree import DecisionTreeClassifier

X, y = load_breast_cancer(return_X_y=True)

search = GridSearchCV(
    DecisionTreeClassifier(random_state=0),
    param_grid={"max_depth": [2, 3, 4, 5, 6], "min_samples_leaf": [1, 5, 10]},
    cv=5,
)
search.fit(X, y)

print("Best settings:", search.best_params_)
print("Best score:", round(search.best_score_, 2))

Best settings: {'max_depth': 4, 'min_samples_leaf': 10}
Best score: 0.93

It tested 5 × 3 = 15 combinations, 5 times each. In a real project, run the search on the training set only, and keep a separate test set for the final check.

How to Fix Overfitting

• Get more data. More examples make it harder to memorise.
• Simplify the model. Lower max_depth, raise min_samples_leaf, use fewer features.
• Use regularisation. Models like Ridge, Lasso and LogisticRegression have a setting (alpha or C) that keeps the model simpler.
• Use ensembles. Random forests overfit less than single trees.
• Remove leaky or useless features.

How to Fix Underfitting

• Use a more powerful model (for example, a random forest instead of linear regression)
• Add better features that really explain the label
• Reduce regularisation, or allow a deeper tree

Exercise