Gradient Boosting — StatQuest Notes
Four companion notes for Josh Starmer’s StatQuest series on gradient boosting. Read them in the video order: first see the algorithm work, then derive its formulas.
Each lecture keeps its worked example, with labeled trees, plots, and links to the relevant video moments.
The four lectures
Why predicting residuals improves regression, shared leaf corrections, learning rate, and the sum of trees.
Derive mean initialization, negative gradients, terminal regions, mean leaf corrections, and regression updates.
Log-odds, sigmoid probabilities, classification pseudo-residuals, leaf corrections, and sequential tree updates.
Derive binary log loss, y minus p, initial log-odds, and Newton leaf values from a quadratic approximation.
Follow the examples in order
Lectures 1 and 2 predict weight. Lectures 3 and 4 predict whether someone likes a movie. The derivation lectures use smaller data sets to make every leaf calculation visible.
The figures are redrawn from the teaching examples. Calculations retain precision where the videos round; the notes say when that changes a displayed number.