XGBoost — StatQuest Notes
Follow Josh Starmer’s StatQuest XGBoost series in order: regression, classification, the mathematics, and the speedups. A fifth note accompanies the separate Python walkthrough.
Start with the four-row examples. Watch the tree group rows that need similar corrections, calculate each leaf’s correction, and add a small part of it to the existing prediction.
The complete reading order
Residuals, similarity scores, split gains, pruning, and regularized leaf corrections in one four-row example.
Build a classification tree, calculate cover, and turn regularized log-odds corrections into probabilities.
Derive the Newton leaf value and split score from loss, curvature, and regularization.
Approximate split search, weighted quantiles, learned missing-value paths, parallel work, and memory access.
An executable churn workflow with missing values, one-hot encoding, cross-validation, early stopping, and tree inspection.
Before you start
If predicting a residual still feels strange, read why a tree predicts the remaining correction first. These notes then focus on what XGBoost changes: the split score, regularized leaf values, and efficient tree construction.
Lecture calculations use their stated settings, including an initial prediction of 0.5 and, initially, zero leaf regularization. These are teaching choices. The Python note states its own settings and includes a complete runnable script, measured results, and source data links.