Random Forest

Random Forest combines many decision trees trained on different samples and different subsets of variables. Instead of relying on a single branch, it gathers many small voices and decides by majority.

Watch the forest vote

Random forest lab

Bagging and tree voting

New sample

X: incomeY: debt / income

Ensemble

Number of trees

15

Max depth

4

Voting

Approved

8 of 15 trees vote “approved” (53%).

Each tree’s vote

• −
• +
• −
• −
• −
• +
• −
• −
• +
• −
• +
• +
• +
• +
• +

Single tree (test)

82.8%

Forest (15)

87.8%

• Approved
• Rejected

What to look for

• Each tree trains on a bootstrap sample and a single candidate feature per split.
• A single tree has a staircase boundary; averaging many trees smooths it and reduces variance.
• Accuracy is measured on 400 new cases that no tree has seen.

Key intuition

A single tree is unstable: a small change in the data can alter its structure a lot (high variance). The forest reduces this problem by averaging trees that differ from one another. To achieve this, each tree is trained with:

• a bootstrap sample of the rows (with replacement), and
• a random subset of variables at each split.

Averaging weakly correlated trees lowers the variance without increasing the bias much. In addition, the rows left out of each bootstrap sample (out-of-bag) serve as free validation.

What to look for

• Each tree uses similar rules, but not identical ones.
• The final decision depends on the aggregation, not on a single tree.
• You gain robustness at the cost of losing a simple global explanation.

Applications

A strong baseline for tabular classification, fraud detection, credit risk and diagnosis when you want good performance without much fine-tuning.