Gradient Boosted Machines from Scratch in C++
This implementation series builds Gradient Boosted Machines in plain C++ using only the standard library. It continues the decision-tree implementation path: first a CART regression tree, then an additive ensemble that repeatedly fits trees to gradient targets. The theory lives in Machine Learning · Boosting · Gradient Boosting; this series turns that theory into readable code.
The earlier Boosting page introduced gradient boosting and AdaBoost together. This sub-series separates GBM into a dedicated path, so regression, binary classification, regularization, and validation can be handled with the same level of detail as the decision-tree posts.
Implementation Roadmap
Regression GBM
Build the core additive model for squared-error regression.
- Start from the mean target.
- Fit shallow CART trees to residuals.
- Add each tree with a learning rate.
- Report train and test RMSE as trees accumulate.
Binary Classification GBM
Convert the same loop into a logistic classifier.
- Maintain logits rather than probabilities.
- Use pseudo-residuals $y-p$.
- Return calibrated probabilities with a sigmoid.
Regularization and Evaluation
Add the practical controls that keep boosting honest.
- Learning rate, tree depth, minimum leaf size, and row subsampling.
- Validation curves and early stopping.
- Permutation importance for model inspection.
Shared Pieces
All posts use the same small vocabulary:
Matrix, avector<vector<double>>feature matrix.RegTree, the CART squared-error tree from Regression Tree (CART).learning_rate, the shrinkage factor applied to every tree.trees, a vector storing the fitted correction trees in order.