What is random forest?
- Machine Learning course
- Intermediate
- Lesson 35/112
- 5 min
Random forest is an ensemble algorithm that combines multiple decision trees.
What is a random forest?
A single decision tree learns very different rules as the data changes (high variance). A random forest trains hundreds of trees on different data samples and different feature subsets, and combines their decisions by voting. The trees' errors cancel each other out.
from sklearn.ensemble import RandomForestClassifier
rf = RandomForestClassifier(n_estimators=200, random_state=0).fit(X_train, y_train)
print(rf.score(X_test, y_test))
print(len(rf.estimators_)) # 200 separate treesIt needs no scaling, works well with little tuning and is a strong first choice for most tabular data.
Why do the trees differ?
There are two sources of randomness:
- Bootstrap: each tree is trained on a sample drawn from the training data with replacement.
- Feature subset: at each split only a randomly chosen
max_featuresfeatures are considered (√p by default in classification).
The second matters: if one strong feature were at the root of every tree, the trees would look alike and averaging would help less.
Practice Tree and forest
Train a single DecisionTreeClassifier(random_state=0) and a RandomForestClassifier(n_estimators=200, random_state=0). Print both test accuracies with round(..., 3) on separate lines.
Starter code
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
from sklearn.tree import DecisionTreeClassifier
# buraya yazSample solution
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
from sklearn.tree import DecisionTreeClassifier
agac = DecisionTreeClassifier(random_state=0).fit(X_train, y_train)
rf = RandomForestClassifier(n_estimators=200, random_state=0).fit(X_train, y_train)
print(round(agac.score(X_test, y_test), 3))
print(round(rf.score(X_test, y_test), 3))Expected output
0.902 0.944
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 Which problem does a random forest reduce compared with a single tree?
- It always zeroes bias
- Training time
- Memory use
- Variance (instability) Correct answer
Explanation: The average of trees that differ from each other smooths out a single tree's fluctuations.
2 Create a forest of 300 trees.
rf = RandomForestClassifier(____=300, random_state=0)rf = RandomForestClassifier(n_estimators=300, random_state=0)Explanation: n_estimators is the number of trees in the forest.
3 Match the concepts.
| Term | Meaning |
|---|---|
bootstrap |
Sampling with replacement |
max_features |
Candidate features per split |
estimators_ |
Trained trees |
oylama |
Combining decisions |