Machine Learning · Boosting · Series Guide

Boosting: Build a Model One Correction at a Time

October 2026 · 3 min read

Bagging trains models independently and averages their predictions. Boosting builds an additive ensemble $H(x) = \sum_{t=1}^{T} \alpha_t h_t(x)$ sequentially, choosing each new weak learner $h_t \in \mathcal{H}$ to minimize the loss left behind by the existing ensemble:

$$h_t = \arg\min_{h \in \mathcal{H}} \mathcal{L}\bigl(H + \alpha h\bigr).$$

Fit a learner using the current state, update predictions or weights, and use that updated state for the following learner. — Each round fits a weak learner to the negative gradient of the current ensemble’s loss.

Core derivation

Boosting as Gradient Descent in Function Space

Derive $\arg\min_{h \in \mathcal{H}} \mathcal{L}(H + \alpha h) = \arg\min_{h \in \mathcal{H}} \bigl\langle \frac{\partial \mathcal{L}}{\partial H}, \alpha h \bigr\rangle$ via Taylor expansion and complete the square to get $\arg\min_{h \in \mathcal{H}} \sum_i \bigl(h(x_i) - t_i\bigr)^2$.

Subseries

AdaBoost

Specialize $\mathcal{L}(H) = \sum_i e^{-y_i H(x_i)}$ to exponential loss: derive weighted classification error $\epsilon_t$, optimal vote $\alpha_t = \frac{1}{2}\ln\frac{1-\epsilon_t}{\epsilon_t}$, and multiplicative example reweighting.

Gradient Boosting

Worked regression and classification trees: mean initialization, negative gradients ($y - F$ and $y - p$), leaf averaging, and Newton log-odds updates.

XGBoost

Second-order regularized objectives, similarity scores, split gain, pruning, system optimizations, and an end-to-end Python workflow.