Bagging & Random Forests: Interview Workshop

Answer: They can fit flexible boundaries but change sharply under small training changes. Bootstrap samples create alternative fits, and averaging reduces the unshared part of their prediction variance. Shared bias and shared fluctuations remain. A stable base learner offers less room for this benefit. Review instability.

Answer: It adds a fresh candidate-feature subset at each split. This may stop one dominant feature from controlling every tree and reduce correlation. It may also weaken splits, so feature count is a tuning tradeoff. Sampling features once per tree is a random-subspace construction. Trace the nodes.

Answer: No. More trees stabilize the average. They do not automatically fix brittle leaves, leakage, a distribution shift, or shared bias. Diagnose train and validation behavior; compare leaf size and feature count, then increase trees until gains justify cost. Avoid claiming finite validation scores must improve monotonically.

Answer: Separate training jobs share the original observations, features, labels, and learning rule. Independence of training execution is not statistical independence across repeated datasets. Name what is random before using an independence claim. Conditional versus repeated-data variability.

Answer: The ensemble variance is 1.3 and its limiting floor is 1. With the equal-variance/equal-correlation assumptions, $4[0.25+0.75/10]=1.3$. As the number of trees grows, the limit is $4(0.25)=1$. Increasing to 100 gives 1.03, so most of the remaining variability is shared. This is variance at a fixed input, not the total test MSE.

Answer: The expected omitted fraction is $81/256\approx0.3164$, or 31.64%. A particular row is missed by each draw with probability 3/4. All four draws miss it with probability $(3/4)^4=81/256\approx0.3164$. The expected number omitted is $4(81/256)=1.265625$. The 36.8% rule is a large-$n$ limit, not the exact answer here.

Answer: The row prediction is (2+6)/2=4, so its squared error is zero. Averaging the separate tree errors would give 4 and evaluate the wrong predictor. First aggregate eligible predictions per row; then apply the loss; then aggregate row losses.

Answer: Specify the convention. Hard labels give two positive votes and a positive majority. Probability averaging gives 1.03/3 ≈ 0.3433 and a negative label at threshold 0.5. Never switch aggregation conventions halfway through evaluation.

Answer: For each row, average predictions only from trees whose training samples omit that row. Compute the squared error of that average, then average these errors over covered rows. Coverage is the fraction of rows with at least one eligible tree. This implementation uses the worked example’s one-based row IDs, fitted model dictionaries, and predict(model, x) function:

def evaluate_oob(X, y, models):
    used_rows = [set(model["sample"]) for model in models]
    predictions, counts = [], []
    for row_id, x in enumerate(X, start=1):
        eligible = [
            predict(model, x)
            for model, used in zip(models, used_rows)
            if row_id not in used
        ]
        counts.append(len(eligible))
        predictions.append(
            sum(eligible) / len(eligible) if eligible else None
        )

    errors = [
        (actual - estimate) ** 2
        for actual, estimate in zip(y, predictions)
        if estimate is not None
    ]
    coverage = len(errors) / len(y) if len(y) else 0.0
    mse = sum(errors) / len(errors) if errors else None
    return predictions, counts, coverage, mse

For the worked example, this returns predictions [2, 6, 2.25, 6, 4], counts [1, 1, 1, 1, 1], coverage 1.0, and MSE 6.6125. A row with no eligible tree receives None; it is excluded from MSE and lowers coverage. If no rows are covered, MSE is None.

Answer: There is no valid threshold separating observed values. Return a single constant leaf: the mean target for squared loss, or weighted class proportions for probability prediction. Do not create empty leaves or divide by their zero count. Process tied feature values together when searching valid boundaries.

Answer: Check whether customers have multiple rows. A tree can omit one visit but train on another visit from the same customer. Use entity-based splits for unseen-customer evaluation, check feature availability at prediction time, and inspect learned preprocessing. Then compare a simple baseline and subgroup performance on the corrected split. Match evaluation to deployment.

Answer: Not on the basis of impurity importance alone. First check whether it leaks chronology or labels. Many candidate thresholds can inflate training impurity importance. Evaluate held-out performance with and without it, and inspect held-out permutation importance with repeated shuffles. A predictive association still does not establish a stable or causal relationship.

Answer: No. Ordinary regression forests average leaf target means, so they do not extrapolate beyond the observed training-target range. More trees preserve that structural limitation. Examine the extrapolation need, features, and an appropriate baseline such as a model with an explicit trend.

Answer: Fit and validate a model with an explicit trend, optionally adding a forest to correct its residual errors. Distinguish a larger input, such as floor area, from a larger target, such as price. A larger house does not necessarily cost more than every training house, but an ordinary forest cannot predict above the largest training price even when that is warranted.

1. Start with a model that can continue a justified trend. Fit a linear or ridge regression, or a relationship supported by domain knowledge. A positive fitted coefficient for floor area lets predicted price keep increasing as area increases. The assumption is that this relationship continues into the new range; the ability to output a larger number alone does not establish accuracy.
2. Optionally use a forest to correct the trend. Fit the trend $g(x)$, calculate training residuals $r_i=y_i-g(x_i)$, then train a forest $h(x)$ to predict those residuals. Combine them as:
   $$\hat y(x)=g(x)+h(x).$$
   The trend supplies the extrapolation; the forest learns remaining patterns such as location effects. The combined prediction can exceed the original price range because it adds a changing trend to the residual correction. That correction still needs to remain appropriate in the new region. Regression-enhanced random forests formalize this approach.
3. Evaluate the range you need to support. For larger houses, train on smaller floor areas and hold out larger ones. For future prices, use a time-based split. A random split usually mixes these ranges and can hide this failure. Fit preprocessing, the trend, and the residual forest using only each split’s training portion. Compare the ordinary forest, the trend alone, and the combination on the held-out range; keep the correction only if it helps.

Another option: local linear forests use forest-derived weights to fit local slopes instead of only local averages. Those slopes can extend predictions, but their validity outside the observed region remains an assumption.

Collect examples from the intended deployment range when possible and monitor inputs outside the validated range. If the trend is not defensible there, restrict the supported range or flag the prediction for review. More trees cannot supply evidence about how prices behave in an unobserved region.