Encoding Categorical Features

September 2026 · 10 min read

A categorical encoder decides what relationships a model can use between names. It can preserve separate identities, express an order, share an effect across several names, or replace a name with a statistic. The useful question is therefore “What should these numbers mean to this estimator?”

This overview connects the choices. Each method has its own worked example, figure, implementation, and references. Start with the three foundations below, then follow the branch that matches your feature; you do not need every alternative for every dataset.

One row can need several different encoders

Consider a support request with channel = email, priority = high, and hour = 23. Channel names have no natural order. Priority has the domain order low, medium, high. Hour repeats every 24 hours. Giving all three fields arbitrary integer IDs would hide these differences.

The example uses a full one-hot block for channel, an explicitly ordered code for priority, and a sine/cosine pair for hour. The method chapters explain the assumptions behind each choice.

Start with target labels and ordered inputs

Label encoding is the foundation for class-name IDs. A classifier may use cat = 0, dog = 1, and fox = 2 as target labels. Those IDs select classes; they do not assert that fox is twice dog. Save the mapping so predicted IDs and probability columns decode to the correct names.

Ordinal encoding is appropriate when an input has an intended order. Supply that order explicitly. Codes 0, 1, and 2 communicate order, but a linear model with one coefficient also treats adjacent code gaps as equal. Ordered codes do not by themselves force predicted outcomes to increase with the category.

Preserve identities when names have no order

One-hot encoding gives each retained category its own indicator column. In the request example, email activates the middle channel column. A linear model can learn a separate channel adjustment, subject to its intercept and regularization conventions.

Keep two practical details in view: a wide one-hot matrix can be stored sparsely, and an all-zero block for an unknown name is not a learned unknown effect. Dropping a reference category can also make that reference and an unknown input look identical. The chapter works through both cases.

Integer IDs are not automatically harmless for trees. A numeric threshold tree can only split contiguous ranges of the supplied IDs at one node. Changing the IDs can change which category groups are easy to isolate; the label-encoding figure shows this directly. An estimator’s explicit categorical-feature support is a different interface from treating an integer column as numeric.

With many categories, decide what can be shared

Cardinality means the number of distinct category values. A large vocabulary creates both storage costs and a statistical problem: some names may have too little support for reliable individual effects. These alternatives make different compromises:

Compact output is not the same as retaining the same information or model flexibility. Compare the whole pipeline on the intended task, including rare and unseen inputs.

If labels help define the feature, protect both boundaries

Target encoding uses a category’s target statistics, usually with smoothing. Weight of evidence compares category prevalence within two target classes. Both can produce compact inputs, but both use labels.

Keeping outer validation labels out is necessary. Training rows also need features that do not reveal their own labels. Use cross-fitting inside the outer training set, with entity or time exclusions when the task requires them:

A plain supervised encoder’s fit_transform name does not guarantee cross-fitting. Scikit-learn’s TargetEncoder.fit_transform implements it; other encoders may simply fit and transform the same rows. Check the specific implementation and its supported split strategy.

Define the prediction-time contract

Before fitting the model, decide which distinctions future inputs must retain:

A valid output shape does not mean the model has learned how to handle that case. Inspect the values an unseen or missing input actually produces. Save encoders with the model and preserve column order; a prediction batch must not redefine the representation.

Choose a reading path

• First pass: label IDs → ordered inputs → separate category indicators. These establish the difference between identity, order, and numeric spacing.
• A large vocabulary: start with rare grouping, then compare prevalence features, fixed-width hashing, and bit patterns for the constraint you actually face.
• Use label statistics: read target encoding and cross-fitting before WOE for binary outcomes.
• A repeating feature: go to sine/cosine encoding for hours, weekdays, and other known cycles. Its circular geometry is a separate idea from compressing arbitrary names.

Evaluate the representation together with the estimator

Use one permitted split to compare candidate pipelines, then inspect overall quality, rare/unseen cases, and memory costs. If deployment faces new entities, use the appropriate group split; if it predicts the future, use a time-based split. Encoding choices cannot repair a misleading evaluation boundary.

The method chapters contain runnable Python examples and small C++ implementations of the key operations. Continue to handling missing data when the next question is how to represent absent observations.

References and further reading

• Scikit-learn: categorical encoding guide — input representations, indicator columns, and infrequent categories.
• Scikit-learn: TargetEncoder — the distinction between cross-fitted training features and later transformations.
• Scikit-learn: FeatureHasher — fixed-width features, signed hashing, and stateless transformation.