Feature Engineering
Feature engineering decides how available information is presented to a model. A raw timestamp becomes an hour-of-day pattern, an event history becomes a count, and a message becomes a vector of word weights. The aim is to expose useful structure while preserving what each observation means.
This series follows six common transformations through small worked examples, figures and optional code. Use this overview to choose a representation; use the individual chapters to understand its calculation and limits.
Start with the prediction you actually need to make
Suppose a delivery service predicts an order's eventual delivery duration when the order is placed. At 10:00 UTC, the request includes a 2 kg parcel, an express-service choice, a customer ID and a short message. Past orders may already be available. Actual arrival time is still unknown.
That description establishes the row: one prediction for one order at its placement time. It also establishes the information boundary. A column can be highly predictive and still be unusable if it only becomes known after that time.
Express = yes
Request time = 10:00 UTC
Customer history and message
Weight × express = 2
Calendar or cyclic time
Historical counts and text weights
Clean the source fields first. “2 kg” and “2 lb” cannot become comparable features by merely dropping the unit string. A customer ID with leading zeros must keep its identity. The data-cleaning series establishes those contracts before this series constructs model inputs.
Choose the chapter by the structure you want to expose
Use extra powers and products when a straight line misses smooth curvature. Inspect the expanded columns and compare fits that underfit, follow the relationship or memorize noise.
Let the effect of one input depend on another. Economy and express prices make changing slopes visible; a separate example explains meaningful ratios and zero exposure.
Represent interval-level behavior. Compare equal-width, quantile and k-means boundaries, then separate bin assignment from one-hot encoding and inspect the information lost.
Separate calendar position, repeating cycles and elapsed time. A clock diagram explains sine/cosine features, while time-zone examples show why the local calendar matters.
Turn eligible histories into counts, totals and means. Move a prediction cutoff to expose delayed availability, rolling windows, empty groups and join cardinality.
Follow three documents from a fixed vocabulary to counts, document-frequency weights and normalized vectors. Inspect unknown words and the limits of a lexical representation.
Start with polynomial features if your question is how a linear model can bend. Start with datetime and aggregation if the data contains histories or recurring activity. For messages or documents, the text chapter is self-contained after basic text normalization.
Write the feature's meaning before writing its code
A useful feature name should have an equally clear definition. “Customer count” is ambiguous; “number of distinct orders for this customer in the 24 hours before the prediction, available by the cutoff” specifies a reproducible quantity.
| Feature decision | What to specify |
|---|---|
| Measurement | Quantity, unit and valid range. A squared measurement has squared units. |
| Category or text | Normalization, tokenization, encoding and treatment of unseen values. |
| Time | Business time zone, cutoff and whether the feature represents a duration or a cycle. |
| History | Entity key, window, source availability and denominator used by each statistic. |
| Missing input | Whether absence means zero, unknown, no history or a source failure. |
Keep the original observations or their provenance where needed to inspect these choices. An engineered feature is a view of the source, not a replacement for understanding it. For example, a ratio of 0.2 may be supported by 10 observations or 1,000; the denominator can remain useful alongside the ratio.
Separate fixed transformations, fitted parameters and changing histories
A fixed transformation follows a chosen rule, such as multiplying weight by an express indicator. Once its inputs and definition are fixed, it has no mean or vocabulary to learn from the dataset. Choosing whether to use it is still a model-development decision.
A fitted transformation estimates parameters from data, such as scaling statistics, quantile-bin edges, a vocabulary or IDF weights. Fit these using the training portion of each validation fold. Save them with the model and reuse them for new observations.
A historical feature changes because new events become available or old events leave its window. Reconstruct it at each prediction's cutoff. Fitting the downstream model only on training rows does not repair a history that was built using future events.
The pipeline chapter explains how to keep fitted transformations within validation. The aggregation chapter covers chronological history reconstruction. Both boundaries matter when a model uses historical features and a fitted scaler or encoder.
Evaluate the representation with the model that will use it
A feature that helps one model may add little to another. Products can give a linear model an interaction it could not otherwise express. A tree may already learn a related split structure. Binning can simplify a noisy relationship but remove distinctions another model could use.
Compare a time representation as a family, for example, rather than attributing a gain to one sine column while forgetting the matching cosine. Inspect behavior on new entities, later dates and rare categories when those are part of deployment. Also consider latency and source dependencies: a modest predictive gain may require an expensive or unreliable lookup.
When a candidate fails, examine what assumption was wrong. Perhaps the relationship is not smooth, the history window misses useful activity, or most new messages contain unseen vocabulary. Those findings should guide the next representation rather than simply adding more columns.
Keep the fitted representation reproducible
Save feature names and order, units, encoding rules, fitted parameters, time semantics and missing-value policies with the model. Training and serving must interpret each column the same way. The chapter examples make these contracts explicit through small calculations you can reproduce.
Once the candidate features exist, feature selection asks which columns to retain. Dimensionality reduction may instead build a smaller representation from combinations of inputs. These decisions should serve the same prediction task and validation design.
References and further reading
- Scikit-learn: common pitfalls — consistent preprocessing and validation boundaries for fitted transformations.
- Featuretools: handling time — feature definitions tied to cutoff times and information availability.
- Scikit-learn: PolynomialFeatures — explicit expansion of model inputs and its growing feature count.
- Scikit-learn: text feature extraction — vocabulary-based representations and the tradeoffs in their construction.