Remove, Cap, or Keep

September 2026 · 12 min read

A detector identifies observations worth examining. Choosing a treatment requires a second judgment: what does the observation represent, and what population should the model serve? The right response may be correction, retention, capping, or exclusion.

The introductory chapter separates unusualness, error and influence. This chapter turns that distinction into a decision and a fair evaluation. It does not require a new detector formula.

Choose an action from evidence about the row

Context can change the decision. An unusually long request may be a valid report job, a unit error, or a real performance failure. The detector’s score alone cannot distinguish these explanations.

Keeping a row does not require giving it unlimited influence

You can change a model’s sensitivity while retaining the observation. For example, Huber regression uses a loss that becomes linear for large scaled residuals, reducing their influence relative to squared error. It acts on prediction errors, which differ from unusual raw feature values.

A robust loss, feature transformation, contextual model and cap address different problems. Choose according to the feature relationship and error mechanism, then compare performance. A robust loss does not make every kind of extreme feature harmless; scikit-learn’s robust regression guide distinguishes outliers in features from outliers in outcomes.

Hold the evaluation population fixed

Consider a model that always predicts the mean of its training outcomes. Training values are [1, 2, 3, 4, 20], and 20 is a valid rare outcome. Compare two training choices:

• Keep all: the prediction is the mean, 6.
• Remove the largest training outcome: the prediction becomes 2.5.

Now evaluate on three separate held-out observations with outcomes [2, 3, 20]. Let $y_i$ be the observed outcome, $\hat y_i$ its prediction and $n$ the number of evaluation rows. Mean squared error, or MSE, averages their squared differences:

On all three held-out rows, the keep-all model has MSE $[16+9+196]/3\approx73.6667$. The filtered-training model has MSE $[0.25+0.25+306.25]/3=102.25$. The latter helps the two ordinary cases but does worse on the valid rare outcome.

The ranking depends on the evaluation population. Reporting only the remaining ordinary cases hides the cost on rare cases the task still includes. Report the full-workload metric and useful subgroup metrics together.

When the deployed system deliberately rejects or routes some inputs, evaluate that complete system: how many inputs are handled, what happens to the rest, and how well each route works. A metric on accepted inputs alone needs its coverage alongside it.

Put learned treatment decisions inside the training boundary

A fixed parsing correction based on a known schema does not learn from the data. A percentile bound or anomaly detector does. Scikit-learn’s data-leakage guidance explains the train-only fitting boundary; the pipeline notes show how to preserve fitted transformations.

Feature capping at prediction time should also be applied to held-out features during evaluation. Removing training rows is a different operation: it does not imply that corresponding kinds of held-out rows disappear. Eligibility exclusions belong to an explicitly defined task or routing policy.

Reproduce the evaluation comparison

The entire example is arithmetic, so the source of each result is visible. A constant predictor lets us isolate the effect of changing training rows and evaluation rows without introducing another model.

import numpy as np

train = np.array([1., 2., 3., 4., 20.])
predictions = {"keep": train.mean(), "remove": train[train != 20].mean()}
workloads = {"full": np.array([2., 3., 20.]),
             "ordinary": np.array([2., 3.])}
for name, y in workloads.items():
    for treatment, prediction in predictions.items():
        mse = np.mean((y - prediction) ** 2)
        print(name, treatment, f"{mse:.4f}")

full keep 73.6667
full remove 102.2500
ordinary keep 12.5000
ordinary remove 0.2500

The rule train != 20 is an explicit choice in this synthetic example, not a proposed production detector. In real work, record the reason and row identifiers behind a removal policy rather than hiding the decision inside an unexplained filter.

C++: evaluate the same predictions on a stated set of rows

#include <iomanip>
#include <iostream>
#include <stdexcept>
#include <vector>

double mse(const std::vector<double>& outcomes, double prediction) {
    if (outcomes.empty()) throw std::invalid_argument("evaluation rows required");
    double total = 0;
    for (double y : outcomes) {
        const double error = y - prediction;
        total += error * error;
    }
    return total / outcomes.size();
}
int main() {
    std::cout << std::fixed << std::setprecision(4);
    for (double prediction : {6., 2.5})
        std::cout << "prediction=" << prediction
                  << " full=" << mse({2,3,20}, prediction)
                  << " ordinary=" << mse({2,3}, prediction) << "\n";
}

prediction=6.0000 full=73.6667 ordinary=12.5000
prediction=2.5000 full=102.2500 ordinary=0.2500

Leave a treatment decision someone else can inspect

Keep the original data or a recoverable source, the rule and its fitted parameters, the changed row IDs, and before/after counts. Save the evaluation workload and results by meaningful subgroup, including how many cases the model still covers.

A treatment is ready to keep when its purpose, information loss and measured effect are clear. If the evidence is inconclusive, retaining the raw observations and the flags keeps the decision reversible while you investigate.

References and further reading

• NIST: detection of outliers — the distinction between erroneous observations, valid unusual values and accommodation through robust methods.
• Scikit-learn: robust regression — different error mechanisms and models that limit residual influence.
• Scikit-learn: data leakage — learning preprocessing and model choices within the training boundary.