When Scaling Matters

September 2026 · 11 min read

Feature scaling changes the units in which a model sees its inputs. Its effect depends on what the model does with those inputs. Distances, optimization steps, and coefficient penalties respond in different ways; “all models need scaling” is too broad.

Start with a change of units you can understand without any fitted scaler. The physical observations stay fixed while a nearest-neighbor calculation changes its answer.

Minutes or hours can change the nearest neighbor

A query has age 40 years and activity duration 50 minutes. Reference A has age 41 and duration 90; reference B has age 60 and duration 51. A is closer in age, while B is closer in duration.

Naive Euclidean distance squares the coordinate differences, adds them, and takes a square root. In years and minutes, A’s distance is the square root of 1² + 40², about 40.012. B’s is the square root of 20² + 1², about 20.025. B wins.

Scaling chooses a distance weighting

Let $x_j$ and $q_j$ be the reference and query values in feature $j$, and let $s_j>0$ be the divisor used for that feature. Distance after scaling is

A larger divisor reduces that feature’s influence on the distance. Subtracting the same fitted mean from both points cancels in their difference; the divisors determine this distance weighting.

Standardization uses training standard deviations as divisors. It is a useful convention when raw units should not set the balance, but unit variance does not mean equal predictive importance. Domain-based weights or another metric may fit the task better.

import numpy as np
from sklearn.neighbors import NearestNeighbors

# Columns: age in years, activity duration in minutes.
reference = np.array([[41., 90.], [60., 51.]])
query = np.array([[40., 50.]])
names = np.array(["A", "B"])
for label, divisors in [("minutes", [1., 1.]), ("hours", [1., 60.])]:
    model = NearestNeighbors(n_neighbors=1).fit(reference / divisors)
    distances, indices = model.kneighbors(query / divisors)
    print(label, names[indices[0, 0]], round(distances[0, 0], 3))

minutes B 20.025
hours A 1.202

The same issue affects Euclidean k-means and kernel methods based on distances. Covariance-based PCA also responds to feature variances: standardizing changes the variance tradeoff it optimizes. A high-variance direction may be meaningful signal, so inspect that decision rather than removing scale automatically.

Regularization is measured in coefficient units

A linear prediction can stay exactly the same after a unit conversion if its coefficient changes accordingly. Suppose a feature is distance in meters and its coefficient is 3. Express the distance in centimeters and use coefficient 0.03 instead.

Ridge, lasso, and regularized logistic models trade data fit against a coefficient penalty. Scaling changes that tradeoff if you keep the same penalty settings. Tune regularization together with the preprocessing; equalizing feature scales is a convention, not a universal definition of a fair penalty.

Optimization can change even when the best predictions do not

Large differences in feature scales can make a gradient-based optimization problem poorly conditioned: one parameter direction needs much smaller steps than another. Rescaling may help a solver take useful steps and reach a solution more reliably. Correlations, initialization, the optimizer, and learning-rate choices still matter.

Separate this numerical issue from a change in the statistical model. Exact, unregularized least squares with an intercept has the same set of fitted predictions under an invertible affine rescaling of its columns. The coefficients change units. A finite iterative solver may follow a different path; a coefficient penalty can change the optimum itself.

Neural networks often benefit from controlled input scales, but scaling alone does not guarantee faster convergence or better generalization. Verify training behavior and held-out performance.

Ordinary numeric trees usually do not need affine scaling

An axis-aligned tree tests a threshold on one feature at a time. A positive affine transformation, such as subtracting a mean and dividing by a positive scale, preserves the ordering and candidate threshold partitions in exact arithmetic. That is why standardization is usually unnecessary for ordinary numeric decision trees and their ensembles.

Choose the operation before choosing the scaler

Column scaling changes the relative units of features. Unit-norm normalization scales each row instead, deliberately removing its overall magnitude. For cosine similarity, multiplying an entire nonzero row by a positive scalar does not change its direction, while scaling its columns differently generally does. “Cosine needs scaling” is therefore not a complete instruction.

Choose a candidate transformation based on the mechanism you need: distance balance, optimizer conditioning, regularization units, or a task-specific notion of direction. The terminology note separates these operations.

Keep the comparison inside the training boundary

A fixed conversion from minutes to hours uses no fitted statistic. A standard deviation, observed minimum, or median is learned from data and must be fitted only on the permitted training rows. Refit the scaler inside each cross-validation training fold; apply its saved parameters to that fold’s validation rows.

Compare the complete pipeline, including model hyperparameters, on the same evaluation task. Monitor both predictive quality and optimization behavior. A changed nearest neighbor or a faster training curve is evidence about a mechanism, not sufficient evidence of better deployment performance.

C++: make the distance divisors explicit

This small calculation reproduces the unit-conversion figure. The divisors are supplied by the caller; the function does not learn a scaler.

#include <array>
#include <cmath>
#include <iomanip>
#include <iostream>
#include <stdexcept>

using Row = std::array<double, 2>;
double distance(const Row& a, const Row& b, const Row& divisors) {
    for (double scale : divisors)
        if (!std::isfinite(scale) || scale <= 0)
            throw std::invalid_argument("positive finite scales required");
    return std::hypot((a[0] - b[0]) / divisors[0],
                      (a[1] - b[1]) / divisors[1]);
}
int main() {
    const Row query{40, 50}, a{41, 90}, b{60, 51};
    std::cout << std::fixed << std::setprecision(3);
    for (double activity_scale : {1, 60}) {
        const Row divisors{1, activity_scale};
        std::cout << "divide activity by " << activity_scale
                  << ": A=" << distance(query, a, divisors)
                  << ", B=" << distance(query, b, divisors) << "\n";
    }
}

divide activity by 1.000: A=40.012, B=20.025
divide activity by 60.000: A=1.202, B=20.000

References and further reading

• Scikit-learn: importance of feature scaling — worked comparisons for nearest neighbors, PCA, and a full prediction pipeline.
• Scikit-learn: StandardScaler — training-based centering/scaling and estimator sensitivity to feature variance.
• Scikit-learn: Ridge — the least-squares objective with an L2 coefficient penalty.