Random Forests: Diversity, Capacity & Practical Tuning
The variance calculation left us with a problem: correlated trees limit the benefit of averaging. Random forests add feature randomness to the row randomness we already understand. This chapter follows that choice from one node to tuning and prediction cost.
Randomize the menu, then choose the best split
At a node, a conventional random forest draws a subset of candidate features. It searches thresholds on those candidates and chooses the best available split. At a child node it draws a new subset. The threshold is still selected by a training criterion; the randomness primarily changes which features compete.
Imagine a housing model with area, rooms, age, and distance. Area is strongly predictive, so bagged trees often split on it first. If a node receives only age and distance as candidates, it must explore another useful explanation. A later node can use area again. Feature subsampling does not delete that column from the entire dataset.
How often does a dominant feature get a chance?
Let $p$ be the number of input features and $m$ the number sampled uniformly without replacement at a node. Each particular feature appears with probability $m/p$. With 20 features and 5 candidates, the dominant feature is available at one quarter of nodes in expectation. Being available does not guarantee it wins.
If there are two useful features among those 20, the probability that a set of five candidates includes neither is $\binom{18}{5}/\binom{20}{5}=21/38\approx0.553$. So more than half the candidate sets miss both. This makes the tradeoff tangible: aggressively reducing the menu may decorrelate trees but also prevent useful splits.
Check: does lowering candidate count always improve the forest?
No. Under the earlier illustration, variance 9 and correlation 0.2 with 100 trees give ensemble variance 1.872. A weaker tree family with variance 16 and correlation 0.1 gives $16(0.1+0.9/100)=1.744$, slightly lower variance, but its bias might rise enough to worsen total error. Validate the forest, rather than optimizing correlation alone.
Keep three sources of randomness distinct
| Method | Rows | Features and thresholds |
|---|---|---|
| Bagged trees | Bootstrap sample per tree | Usually search all allowed features and their candidate thresholds. |
| Random forest | Commonly bootstrap per tree | Sample features at each node; optimize a split on the selected candidates. |
| Random subspaces | Depends on construction | Choose one feature subset per learner, retained throughout that learner. |
| Extra Trees | Bootstrap is optional | Randomize candidate thresholds as well as feature candidates, then compare eligible candidates. |
These descriptions name mechanisms. Configurations matter: disabling bootstrapping or setting every feature eligible changes how diversity is created. Extra Trees can spend less work searching thresholds but may need more trees or different capacity. There is no dataset-independent winner.
Tune one cause of error at a time
| Observation | Reasonable experiment | What to watch |
|---|---|---|
| Both training and validation error are high | Allow deeper trees or smaller leaves; revisit weak features | Useful flexibility may be missing; adding identical weak trees cannot create it. |
| Training error is low; validation error is high | Increase minimum leaf size; limit depth; examine split leakage | Reduce brittle partitions, but check distribution mismatch before blaming capacity. |
| Predictions change with the seed | Increase tree count while plotting a validation or OOB metric | Look for diminishing gains versus runtime; noisy validation scores need repeated evidence. |
| Trees repeatedly use the same strong feature | Try fewer candidate features | Too little feature access can make all trees weak. |
| Training or serving is too expensive | Limit leaves/depth and measure fewer trees | Serving latency depends on visited paths and implementation, not tree count alone. |
A practical search starts with a moderate forest to identify a useful range of leaf size and feature count, then increases tree count until the averaging is sufficiently stable for the budget. Keep the metric and validation split fixed across candidates. If you repeatedly inspect OOB scores to choose settings, OOB is now a model-selection signal; it is no longer your untouched final assessment.
More trees usually stabilize a forest rather than steadily increasing its effective partition complexity in the way deeper individual trees do. Still, do not promise that every finite-tree validation curve is monotonic or that forests cannot overfit. Capacity, feature selection, label noise, and evaluation design all matter.
What is stored, and what does a query cost?
Each tree stores split features, thresholds, child references, and leaf predictions. A new row travels one path per tree. For $B$ trees with path length at most $d$, this takes roughly $O(Bd)$ node visits, ignoring batching and hardware details. Balanced trees can have depth on the order of $\log n$; highly unbalanced trees can be much deeper. Memory follows total stored nodes.
Tree fits can run concurrently because none needs another's outputs. Avoid assigning every tree a full machine's worth of threads inside an outer parallel search: nested parallelism can oversubscribe CPUs and memory. Benchmark the actual training and serving configurations.
Ordinary regression forests predict averages of leaf target means. With nonnegative training weights, these predictions remain within the training-target range. A rising trend outside observed inputs will therefore flatten rather than extrapolate as a linear model might. Classification probabilities are averaged leaf proportions, which should still be checked for calibration.
Scaling, missing values, and categories
Exact numeric splits depend on feature order, so ordinary positive rescaling rarely changes the possible training partitions. Scaling is generally unnecessary for forests. Floating-point precision, approximate binning, ties, and how a threshold is placed between observed values can still affect behavior.
Native missing-value and categorical support varies by estimator and version. Choose a documented route for the estimator you use. A numeric code for a nominal category imposes an artificial order unless the algorithm is explicitly told to treat it as categorical. An encoding learned from labels must be trained without revealing validation labels.
Next: evaluate the whole prediction procedure
You can now explain both the row and feature randomness and match tuning controls to their effects. The next question is whether the resulting model generalizes. The OOB and evaluation chapter starts with the omitted rows already created by bootstrapping, then asks when those rows form a valid evaluation.
Sources and further reading
- Breiman: Random Forests — Forest strength and correlation.
- RandomForestClassifier documentation — Aggregation, settings, and supported input handling.
- scikit-learn forest guide — Forest and randomized-tree variants.