Decision Tree

A decision tree splits the data space into branches by asking “if-then” questions. Each node tests a condition and each leaf gives a final decision.

Follow a decision path

Decision tree lab

Splits, impurity and leaves (CART)

New sample

X: incomeY: debt / income

Hyperparameters

Max depth

3

More depth fits the training data better, but can overfit.

Min samples per leaf

3

Customer path

Approved

Leaf with 6 of 9 approved (P = 0.67), at depth 3.

Train accuracy

94.4%

Test accuracy

80.3%

Leaves

7

Learned tree

• Income ≤ 59.5n=90 · gini 0.49
  • Income ≤ 45.9n=52 · gini 0.23
    • Debt ≤ 25.1n=37 · gini 0.05
      • Rejected1/5 approved · gini 0.32
      • Rejected0/32 approved · gini 0.00
    • Income ≤ 50.6n=15 · gini 0.48
      • Approved6/6 approved · gini 0.00
      • Rejected0/9 approved · gini 0.00
  • Debt ≤ 79.1n=38 · gini 0.30
    • Income ≤ 68.7n=33 · gini 0.17
      • Approved6/9 approved · gini 0.44
      • Approved24/24 approved · gini 0.00
    • Rejected1/5 approved · gini 0.32

• Approved
• Rejected
• Active path

What to look for

• Each split minimizes the weighted Gini impurity of the two children.
• Increase depth: training accuracy goes up, test accuracy plateaus or drops.
• Move the customer near a threshold and watch how the leaf changes.

How a tree thinks

At each node, the tree tries thresholds on the variables (income, debt, score…) and keeps the split that leaves the leaves as “pure” as possible. Purity is measured with Gini impurity or with entropy:

where is the proportion of class in the node. Both are 0 when the node is pure.

What to look for

• A single variable can send a case down a different branch.
• The main advantage of a tree is that it is easy to explain.
• If it grows too deep, it can memorize the training data and overfit.

Applications

Used in credit scoring, medical rules, customer segmentation, and any setting where explaining the decision matters almost as much as accuracy.