Outlier Detection and Treatment
A suspicious observation creates two separate questions: why does it look unusual, and what should we do with it? A distance or density score can help with the first. It cannot establish that a measurement is wrong or that removing it will improve a model.
This series connects those questions. Start with the meaning of an outlier, learn a few ways to flag candidates, then evaluate a treatment. The method chapters contain the calculations and code; this overview helps you choose a route through them.
Start with the comparison you actually need
A request taking one second may be ordinary for a report generator and unusual for a cached lookup. Pooling both services can hide that distinction. Before choosing an algorithm, identify the population a row should be compared with: the same service, season, device type or operating condition.
Also check what the recorded value means. A parsing error, a unit mismatch and a valid rare event call for different responses. The first chapter separates data quality, statistical unusualness and influence on a fitted model.
Read by question, then follow the calculation
| Question | Chapter and worked example |
|---|---|
| Unusual relative to what? | What is an outlier? Compare the same latency against two service contexts; see a pair of ordinary feature values form an unusual combination. |
| How far is a value from a fitted center? | Z-scores Watch an extreme value inflate the standard deviation when it helps define its own reference. |
| Can I use the middle of the data as a reference? | IQR fences Calculate quartiles, distinguish fences from box-plot whiskers and move the fence multiplier. |
| Which points are easy to isolate across several features? | Isolation Forest Separate raw scores, thresholds and labels; see why a more extreme input need not get a more extreme score. |
| Is a point sparse compared with its own neighbors? | Local Outlier Factor Click among four points and trace neighbor radii, reachability distances and density ratios. |
| What exactly changes when I cap a value? | Winsorizing and capping Compare capping with trimming and distinguish count-based from interpolated-quantile bounds. |
| How do I decide whether to remove, cap or keep? | Treatment decisions See why the apparent benefit of removing a training extreme depends on the evaluation population. |
For a first pass, read the introductory chapter, z-scores, IQR fences and treatment decisions. Add Isolation Forest and LOF when combinations of features or different local densities matter. Read capping before implementing it, because similarly named library operations can produce different bounds.
Understand what each score can see
Z-scores and IQR fences inspect one feature at a time. They are easy to explain, but separately ordinary values can form an unusual combination. A skewed distribution also has a naturally long tail; a tail flag is not automatically a fault.
Isolation Forest compares how quickly random partitions isolate observations. Its score is useful for ranking candidates, but it does not explain their cause. Feature representation still determines the available splits.
LOF compares neighboring density estimates. The distance metric, feature scales and neighborhood size determine that comparison. A tight group of abnormal points can support one another and appear locally ordinary.
These methods need not flag the same rows: they encode different notions of unusualness. Inspect disagreements against the task instead of treating agreement among detectors as proof of bad data. The relevant method chapter explains its assumptions before introducing its equations.
Decide whether you are inspecting a batch or scoring future inputs
Exploring a collected dataset and evaluating a predictor are different uses. During model evaluation, learned centers, quartiles, scaling parameters and detector state belong to the training fold. Refitting them on validation or test inputs changes the reference you are testing.
LOF makes this distinction explicit in its API: fit_predict scores the training collection, while novelty=True enables methods for unseen queries. Read the two small examples before substituting one interface for the other. The pipeline series covers the broader fit-and-apply workflow.
Evaluate the decision, not just the cleaned histogram
A plot with a shorter tail does not establish better predictions. Keep a no-treatment baseline and compare candidates under the same split and metric. If valid rare cases will occur in use, include them in evaluation and report their performance separately when useful.
If the deployed system rejects cases or routes them to review, report that coverage alongside performance on accepted cases. Otherwise, a model can appear better simply because its evaluation excludes the difficult observations. The treatment example makes that effect concrete.
Save the original value, the transformed value when applicable, the flag and the rule version. This makes it possible to explain a decision and revisit it when the population or understanding of the data changes.
References and further reading
- NIST: detection of outliers — labeling candidates, investigating errors and accommodating valid unusual observations.
- Scikit-learn: novelty and outlier detection — the scoring interfaces and distinct assumptions behind batch and future-query detection.
- Scikit-learn: data leakage — keeping learned preprocessing within the training data.