Out-of-Bag Evaluation, Leakage & Interpretation

September 2026 · 14 min read

A forest gives every tree a different training sample. Some original rows are missing from each sample. Those omissions let us build predictions for a row using only trees that did not train on it. We will calculate that estimate before deciding when it is trustworthy.

Evaluate each row using trees that omitted it

A row is out of bag for trees whose bootstrap samples excluded it. Combine only those trees’ predictions for that row, then compute its loss. Average the row losses to get the OOB score. With too few trees, some rows may have no OOB prediction.

Fit three stumps and build each row’s eligible committee

Give the five original rows feature values $x=[1,2,3,4,5]$ and targets $y=[1,2,4,4,7]$. We will fit three regression stumps to fixed, possible bootstrap draws. A stump has one split and predicts the training-target mean within each leaf. We count every repeated draw when fitting.

For each stump, try midpoints between adjacent distinct sampled inputs and minimize the sum of squared residuals. Use the explicit routing rule left if $x<t$, right otherwise; if candidate costs tie, choose the smaller threshold. This makes every prediction reproducible, including a new input that equals a threshold.

For T1, the right-leaf targets are $[4,7,7]$, whose mean is 6. Dropping the repeated row would give 5.5 and change the fit. For T3, the left-leaf mean is $(1+2+2+4)/4=2.25$. The split at 4.5 has total SSE 4.75, below the alternative split at 3 with SSE $31/6\approx5.1667$.

The OOB MSE is $(1+16+3.0625+4+9)/5=6.6125$. Here every row has exactly one eligible stump by construction. With more trees, first average all eligible predictions for a row, then square its error. Averaging the separate tree errors instead evaluates a different predictor.

For example, if two eligible trees predict 2 and 6 for a target of 4, their averaged prediction has squared error 0; their separate squared errors average to 4. More generally, average individual squared error equals squared error of the average plus the variance of those predictions.

These five observations and three fixed draws are a transparent arithmetic example, not evidence that bagging improves a real task. A row with no eligible tree must remain unscored until more trees provide coverage; silently giving it a prediction of zero changes the evaluation.

The standard-library reproduction enumerates every stump candidate using exact fractions, records the fit and OOB trace, and regenerates the correlation figure:

python3 blog/ml/bagging/figures/bagging_oob_example.py

OOB is useful for appropriately sampled independent rows. It does not enforce time or entity boundaries, undo preprocessing leakage, or remain an untouched test after repeated tuning. Keep the evaluation design aligned with how predictions will be used.

Inspect the eligible trees for any row

Use the same fitted stumps from the table above. Choose a row and check the sample memberships before reading the result. Changing the row changes the committee; it does not refit any tree.

Count predictions before you average them

For $n$ rows and size-$n$ bootstrap samples, write $q=(1-1/n)^n$ for the probability that a particular row is omitted by a tree. With independently generated samples, its expected number of OOB trees is $Bq$. The probability it receives no OOB prediction is $(1-q)^B$.

For large $n$ and just three trees, this probability is approximately $0.632^3=0.252$ for a given row. With 100 trees, expected OOB committee size is about 36.8. Those trees can still be correlated, so this count is not an effective independent sample size. Our five-row worked example arranged full coverage; random draws do not guarantee it.

oob_sum = zeros(n)
oob_count = zeros(n)
for tree, sampled_ids in fitted_trees:
    for i in original_ids_not_in(sampled_ids):
        oob_sum[i] += tree.predict(X[i])
        oob_count[i] += 1
covered = oob_count > 0
oob_prediction[covered] = oob_sum[covered] / oob_count[covered]
report_coverage(covered)
score(y[covered], oob_prediction[covered])

Keep uncovered predictions explicitly missing. For classification, accumulate aligned probability vectors instead of scalar predictions. The metric is a separate choice: OOB classification accuracy, log loss, and recall at a threshold answer different questions.

Match the split to the question you want to answer

An OOB score evaluates row omission within the observed training population. It is not literally a score from the full forest on a newly collected test dataset: each row uses a smaller, varying subset of trees. Treat it as an estimate with assumptions, rather than insisting it is universally unbiased or interchangeable with cross-validation.

Omission must apply to learned preprocessing too

Suppose you encode each category using the mean target over the whole dataset, then train a forest with OOB scoring. A row omitted by a tree has already contributed its label to its encoded feature. The OOB mask cannot undo that exposure. Supervised feature selection before the forest creates a similar problem.

Putting preprocessing before a forest in a single pipeline helps when the whole pipeline is fit within external training folds. It does not automatically make the forest's internal OOB predictions out of sample for those upstream fitted steps. To evaluate such a pipeline, use external folds respecting the data boundaries, or a construction that fits the learned preprocessing inside each base learner's allowed sample.

Explain importance without overstating it

Impurity importance adds up the weighted training-criterion reductions assigned to each feature across tree splits. A continuous ID or high-cardinality feature has many opportunities to find a chance split, so a large training importance can be misleading.

Permutation importance asks a different question: how much does a fitted model's held-out score worsen when one feature's values are shuffled? Repeat the shuffle to inspect variability. Correlated features can substitute for one another and hide each other's apparent importance; shuffling also creates combinations absent from the real population. Report the metric and dataset, and avoid reading the result as a causal effect.

Separate ranking, probability quality, and the decision

A fraud model can rank cases well while giving probabilities that are too extreme. Check a proper probability loss and reliability when probabilities drive costs; choose the action threshold using validation data and the investigation budget. If you fit a calibration model, its training predictions also need to come from a valid held-out or out-of-fold procedure.

The practice chapter now asks you to connect sampling, model behavior, and evaluation in one answer. That connection matters more than memorizing a default hyperparameter.

Sources and further reading

• Breiman: Random Forests — OOB estimation.
• scikit-learn: Data leakage — Train-only learned preprocessing.
• scikit-learn: Permutation importance — Interpretation and correlated-feature limitations.
• scikit-learn: Calibration — Probability evaluation and calibration.