Machine Learning from Scratch in C++
This series builds machine-learning algorithms from scratch in plain C++, using nothing but the standard library, no NumPy, no scikit-learn, no LibTorch. The goal is working, compilable code that you can read end to end. Every post ships a complete program and reports the numbers it actually prints. The conceptual derivations (entropy, information gain, the bias-variance story) live in the companion Machine Learning · Decision Trees notes; here the focus is implementation.
The series has two halves. The models grow the decision-tree family one idea at a time, each reusing the last. The data preprocessing toolkit covers the steps that come before any model: getting raw data into a clean numeric matrix and evaluating a model honestly.
The Models
Regression
Linear and logistic regression from scratch, from reliable solvers and regularization to diagnostics, classification metrics, and calibration.
- Linear Regression: least squares, iterative solvers, regularization, robust losses, and evaluation.
- Logistic Regression: log-loss, Newton and IRLS solvers, softmax classification, metrics, and calibration.
Decision Trees
The foundation: one recursive tree, then the same skeleton for regression.
- Decision Tree Classifier — Gini impurity, the sorted-sweep best split, recursive growth (100% / 97.2%).
- Regression Tree (CART) — squared-error splits via an incremental sum / sum-of-squares sweep.
Random Forest
Average many decorrelated trees to cut variance.
- Random Forest — bootstrap sampling, feature sub-sampling (
mtry = √D), majority vote, and out-of-bag error.
Gradient Boosted Machines
Add shallow regression trees in sequence, each following the current gradient target.
- Overview: dedicated GBM roadmap, shared tree pieces, and links to the conceptual series.
- Regression GBM: residual fitting, shrinkage, and RMSE curves.
- Binary Classification GBM: logits, sigmoid probabilities, and pseudo-residuals.
- Regularization and Evaluation: subsampling, early stopping, and permutation importance.
Boosting & AdaBoost
Bridge gradient boosting and AdaBoost, then compare sequential boosting with bagging.
- Gradient Boosting & AdaBoost — residual fitting (negative gradient) and reweighted decision stumps, with convergence curves.
Explainability
Explain trained models by perturbing features, fitting local surrogates, and estimating additive attributions.
- Overview: model-agnostic explanation interface and roadmap.
- Permutation Importance: shuffle one column and measure the score drop.
- LIME from Scratch: local perturbations plus weighted linear regression.
- Kernel SHAP from Scratch: coalition masks plus weighted regression.
Data Preprocessing
Real data rarely arrives as a clean numeric matrix with a ready split. The Data Preprocessing sub-series implements the everyday steps from scratch and links each one to the model it feeds.
Data Preprocessing
- Train/Test Split & Shuffling — plain and stratified splits that preserve class ratios.
- Feature Scaling — standardization and min-max, fit on train and applied to test.
- Encoding Categorical Features — label and one-hot encoding.
- Handling Missing Values — mean and median imputation.
- K-Fold Cross-Validation — an honest accuracy estimate over the decision tree.