Feature Scaling
Scaling defines the units in which a model reads numerical inputs. Changing those units can change distances, the effect of a coefficient penalty, or the speed of optimization. Scaling is useful when that change serves the model and task; it does not automatically make predictions better.
This chapter is the map for the feature-scaling series. Start with the effect you want, follow one concrete fit-and-reuse example, then choose a method chapter. The detailed derivations and implementations live in those chapters so you do not need to read the same calculation repeatedly.
Why changing the units can change the result
Suppose a nearest-neighbor model compares people using age and activity duration. A duration difference of 60 minutes contributes 60² to raw squared Euclidean distance. Express the same duration in hours and it contributes 1². The age coordinate did not change, so the model now weighs the two differences differently.
A large numerical range can dominate a distance, but that does not prove the corresponding feature is unimportant or that unit variance is the correct weighting. Scaling chooses a metric. The worked nearest-neighbor example changes units and tracks the neighbor without assuming which answer is right.
Other models react for different reasons. Rescaling a feature changes the coefficient needed to express the same prediction, which changes a penalty on coefficient size. It can also make gradient-descent steps easier to tune across parameters. These are separate effects, developed in When Scaling Matters.
Choose what the transformation should accomplish
Most feature scalers work down each training column: learn one center or divisor per feature, then reuse it. Row normalization works across each example: calculate a divisor from that row’s own coordinates. Decide which axis you mean before choosing an API.
| Method | Reference it uses | What to remember |
|---|---|---|
| Standardization | Training column’s mean and standard deviation | Nonconstant training columns get mean 0 and variance 1. The distribution’s shape remains. |
| Min-max | Training column’s minimum and maximum | Maps training extrema to chosen endpoints. Future values can lie outside them. |
| Robust scaling | Training median and interquartile range | Uses the middle of the distribution to set the scale. Outliers remain in the data. |
| Max-abs | Training column’s largest absolute value | Zero stays zero. A single extreme reference value can set a large divisor. |
| Unit-norm rows | Each row’s own length under the chosen norm | Nonzero rows get unit length. Their original magnitudes are discarded. |
The method chapters handle constant columns, missing inputs, and optional clipping explicitly. “Normalization” alone does not specify one of these operations; use the terminology comparison when a paper or API is ambiguous.
Follow one scaler from fitting to prediction
Use one feature with training values 10, 20, 30, and 40. Their mean is 25. Their population standard deviation is the square root of the average squared distance from 25, approximately 11.1803. A standardizer saves these two numbers.
A future value of 50 then becomes (50 − 25) / 11.1803, approximately 2.2361. It does not change the saved mean or divisor. That is the distinction between fit and transform:
Fit the estimator on transformed training rows, and predict from validation or future rows transformed by that same scaler. A pipeline fitted within each cross-validation fold keeps this boundary intact. Fitting a scaler once on the whole dataset before cross-validation allows held-out rows to influence the representation.
Future data need not have mean 0, variance 1, or values within the training min-max interval. Recomputing statistics merely to force those properties changes the transform that the estimator receives. The standardization code separates learning the statistics from reusing them.
Check what the model actually depends on
| Mechanism | What scaling can change |
|---|---|
| Euclidean distance | Relative contributions of feature differences, affecting neighbors or clusters. |
| Cosine similarity | Column scaling can change direction. Multiplying a whole nonzero row by one positive number preserves direction. |
| Coefficient penalties | The cost of expressing an effect with a coefficient; tune the penalty with the chosen preprocessing. |
| Numerical optimization | Step sizes and conditioning. This can change convergence even when the available prediction functions are the same. |
| Covariance-based PCA | Which directions account for the most variance. Scaling changes the variance objective. |
Ordinary numerical threshold trees usually do not need feature scaling. Positive affine changes preserve the candidate training partitions in exact arithmetic. That does not mean every monotonic transform yields identical predictions on unseen values: threshold placement between observations can differ. The tree counterexample explains the distinction.
Sparse storage is another constraint. Subtracting a nonzero mean can fill formerly implicit zeros. See Scaling Sparse Data before applying default centering to a large sparse matrix.
Choose a reading route
If you are onboarding, read the first two chapters below, then branch according to the representation you need. Each method chapter includes a small calculation, a figure, implementation details, and primary references.
Distances, coefficient penalties, optimization, PCA, and the tree exception.
Fit a z-score transform; distinguish training statistics from future values and distribution shape.
Trace reference extrema, out-of-range inputs, and the information lost through clipping.
Calculate quantiles and the IQR, then see why transformed outliers still remain.
Separate column divisors from row lengths and inspect which information is retained.
Follow implicit zeros, fitted variance, and CSR storage through a transform.
Translate ambiguous names into an operation with an axis, reference, and intended property.
The next data-preparation topic is encoding categorical features. Encoding decides what the coordinates represent; scaling decides how those coordinates are measured.
References and further reading
- Scikit-learn: importance of feature scaling — examples showing effects on distance-based learning and PCA.
- Scikit-learn: preprocessing guide — the separate scaling operations, sparse constraints, and row normalization.
- Scikit-learn: data leakage — fit preprocessing on training subsets and use pipelines during evaluation.