Bagging: From One Unstable Tree to a Forest
A decision tree can change sharply when a few training rows change. A different first split changes the later questions, so the final prediction may move even when the underlying problem has barely changed. Bagging asks: can we stabilize that prediction by averaging several plausible trees?
This path starts with the sampling mechanism and ends with the decisions you need to defend in an interview. You need to know how a tree chooses a split and predicts from a leaf; the tree foundations explain those steps. No probability beyond averages is needed to begin; derivations appear after examples.
One model, then a committee
Imagine training three regression trees that predict 4, 7, and 7 for the same apartment. Their average is 6. The arithmetic is easy. The important question is why those trees disagree: we train them on different resamples of the same training data. That creates variation we can average over, without collecting new labels.
If every tree makes exactly the same mistake, averaging repeats it. If some predict too high while others predict too low, their fluctuations partly cancel. This is the thread connecting bootstrapping, the variance formula, and random forests.
Read in order, then explain it back
- Bootstrap aggregation
Draw samples, specify aggregation, and derive the variance floor. Explain what more trees can and cannot fix.
- Random forests
Randomize features at each node, compare Extra Trees, and connect capacity controls to runtime and error.
- OOB, evaluation, and interpretation
Follow actual fitted stumps, inspect each row's eligible committee, and detect leakage and misleading importance.
- Interview workshop
Use a 45-minute mock interview, numeric answers, and an OOB coding task to find the gaps in your understanding.
Read one chapter per session. Before continuing, reproduce its central example without the table. At the end, move to boosting, where the next tree depends on the current ensemble.
Before opening the next chapter
Try explaining why training the same deterministic tree five times on identical rows does not create five useful independent opinions. Then ask what training with replacement changes. Keep that question in mind as you follow the five-row example.