ML Decision Trees

A decision tree makes predictions by asking a series of simple yes/no questions about the data.

How a Decision Tree Works

Think of how a shopkeeper decides whether to sell on credit: "Is the customer a regular? If yes, has he paid on time before? If yes, give credit." Each question splits the customers into smaller groups.

A decision tree learns these questions from data:

• The top question is the root
• Each question is a node, and each answer leads down a branch
• The final answers at the bottom are leaves

The tree picks the question that best separates the classes at each step. It keeps splitting until the groups are pure or a limit is reached.

Training and Reading a Tree

Here a car dealer looks at past customers to predict who buys a car on instalments, using age and monthly income:

from sklearn.tree import DecisionTreeClassifier, export_text

# [age, monthly income in thousand Rs]
X = [[22, 30], [25, 45], [30, 80], [35, 120], [40, 60],
     [45, 150], [28, 25], [50, 90], [23, 95], [21, 110]]
# Did the person buy a car on instalments?
y = ["no", "no", "yes", "yes", "no", "yes", "no", "yes", "no", "no"]

tree = DecisionTreeClassifier(max_depth=2, random_state=0)
tree.fit(X, y)

print(export_text(tree, feature_names=["age", "income"]))
print(tree.predict([[33, 100], [22, 100]]))

|--- age <= 29.00
|   |--- class: no
|--- age >  29.00
|   |--- income <= 70.00
|   |   |--- class: no
|   |--- income >  70.00
|   |   |--- class: yes

['yes' 'no']

Reading the tree

1. First question: is the person 29 or younger? If yes, the answer is "no".
2. If older than 29: is income Rs 70,000 or less? If yes, "no". Otherwise, "yes".

So a 33-year-old earning Rs 100,000 is predicted to buy, but a 22-year-old with the same income is not. You can explain this model to anyone, which is the biggest strength of decision trees.

Feature Importance

A trained tree tells you which features it used most:

from sklearn.datasets import load_iris
from sklearn.tree import DecisionTreeClassifier

iris = load_iris()
tree = DecisionTreeClassifier(max_depth=3, random_state=0)
tree.fit(iris.data, iris.target)

for name, score in zip(iris.feature_names, tree.feature_importances_):
    print(f"{name}: {score:.2f}")

sepal length (cm): 0.00
sepal width (cm): 0.00
petal length (cm): 0.05
petal width (cm): 0.95

The importances add up to 1. Here petal width does almost all the work. This helps you understand your data and drop useless features.

Controlling the Size of a Tree

If you let a tree grow without limits, it keeps asking questions until it memorises every training example, including mistakes and noise. This is overfitting.

Start with max_depth between 3 and 6 and adjust. The Overfitting chapter shows how to choose it.

Decision Trees for Numbers

Trees can also predict numbers. Use DecisionTreeRegressor instead of DecisionTreeClassifier. Each leaf then holds the average value of the training examples that reach it.

Random Forests

One tree can change a lot when the data changes a little. A random forest trains many trees, each on a random sample of the rows and features, and combines their votes. The result is usually more accurate and more stable.

from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)

tree = DecisionTreeClassifier(random_state=0).fit(X_train, y_train)
forest = RandomForestClassifier(n_estimators=100, random_state=0).fit(X_train, y_train)

print("One tree:", round(tree.score(X_test, y_test), 3))
print("Forest:  ", round(forest.score(X_test, y_test), 3))

One tree: 0.93
Forest:   0.972

n_estimators=100 means 100 trees. Random forests and a related method, gradient boosting, are among the best choices for table data in real companies.

Pros and Cons

Exercise