Machine Learning
What is random forest?
Random forest is an ensemble algorithm that combines multiple decision trees.
What is a random forest?
A single decision tree learns very different rules as the data changes (high variance). A random forest trains hundreds of trees on different data samples and different feature subsets, and combines their decisions by voting. The trees' errors cancel each other out.
from sklearn.ensemble import RandomForestClassifier
rf = RandomForestClassifier(n_estimators=200, random_state=0).fit(X_train, y_train)
print(rf.score(X_test, y_test))
print(len(rf.estimators_)) # 200 separate trees
It needs no scaling, works well with little tuning and is a strong first choice for most tabular data.
Why do the trees differ?
There are two sources of randomness:
- Bootstrap: each tree is trained on a sample drawn from the training data with replacement.
- Feature subset: at each split only a randomly chosen
max_features features are considered (√p by default in classification).
The second matters: if one strong feature were at the root of every tree, the trees would look alike and averaging would help less.
Exercise Tree and forest
Train a single DecisionTreeClassifier(random_state=0) and a RandomForestClassifier(n_estimators=200, random_state=0). Print both test accuracies with round(..., 3) on separate lines.
Show example solution
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
from sklearn.tree import DecisionTreeClassifier
agac = DecisionTreeClassifier(random_state=0).fit(X_train, y_train)
rf = RandomForestClassifier(n_estimators=200, random_state=0).fit(X_train, y_train)
print(round(agac.score(X_test, y_test), 3))
print(round(rf.score(X_test, y_test), 3))
Test yourself
-
Which problem does a random forest reduce compared with a single tree?
- It always zeroes bias
- Training time
- Memory use
- Variance (instability)
Show answer
Variance (instability) — The average of trees that differ from each other smooths out a single tree's fluctuations.
-
Create a forest of 300 trees.
rf = RandomForestClassifier(____=300, random_state=0)
Show answer
rf = RandomForestClassifier(n_estimators=300, random_state=0)
-
Match the concepts.
- bootstrap
- Sampling with replacement
- max_features
- Candidate features per split
- estimators_
- Trained trees
- oylama
- Combining decisions
Next lesson: Ensemble
