Aggregation and Group Features

September 2026 · 15 min read

An aggregation feature summarizes a set of related records for one prediction. A customer's recent order count describes activity; their average order amount describes typical size. Computing a mean is easy. The important work is deciding which records belong in it.

Define four things before writing groupby: the entity being summarized, the prediction time, the history window and when each source record became available. This chapter uses one customer history to make all four visible.

One prediction row can summarize many event rows

Suppose we make a customer-level prediction at 10:00 on 14 September 2026. We want order count, total spend and average amount over the preceding 24 hours. All times in this example are UTC, and every amount is in the same currency.

Count, sum and mean answer different questions. A mean of 30 could describe one order or hundreds, so retaining a count can expose how much history supports it. Resolve duplicate events before aggregating; copying a row should not invent another purchase.

Past occurrence and past availability are separate requirements

At 10:00, include orders for A that occurred at or after 10:00 yesterday and strictly before 10:00 today, and whose information was available by 10:00 today. The left end of the window is included; the prediction-time boundary is excluded. These are explicit choices for this feature.

An order placed at 09:50 but delivered to the feature system at 10:10 is unavailable for the 10:00 prediction. Its event time is in the past, yet using its amount would require future information. Similarly, a correction received later must not silently replace the version an earlier prediction could see. Preserve availability times and versions when that distinction matters.

This is the purpose of reconstructing features as of a prediction time. The Feast discussion of point-in-time joins explains both event-time retrieval and the additional availability constraint needed for late backfills.

Move the prediction time, not the source history

The source below also contains an old order, a late-arriving order, an order exactly at the initial cutoff, a later order and another customer's order. Move the cutoff to see records enter or leave the eligible set.

The resulting (count, spend, mean) triples are (2, 60, 30), (4, 220, 55), (5, 300, 60) and (4, 280, 70). Notice that count decreases at the final cutoff while the mean increases. Aggregation is a summary of a changing set, not a quantity that must always increase with time.

What does an empty history mean?

For a new customer C with no eligible orders, use count 0 and total spend 0, but leave mean amount undefined. Filling the mean with zero would conflate “no observations” with “observed orders averaging zero.” An imputer and a missing-history indicator can give the model an explicit treatment.

That interpretation assumes the history feed is available and the requested window is covered. An outage or missing source partition is not evidence that no orders occurred. Preserve a coverage or freshness signal when the system needs to distinguish those cases.

Also distinguish row count from count of valid amounts. If an order exists but its amount is missing, it can contribute to the order count without contributing to an average amount. Track the denominator actually used. The numerical example assumes every eligible order has a valid amount.

Attach features to predictions without multiplying rows

Customer A can have many prediction requests, each with a different cutoff. An aggregate keyed only by customer is therefore insufficient for historical features. Give each prediction a stable ID and produce exactly one feature row for that ID.

For a fixed reference table with one row per entity, a many-to-one join may instead be appropriate. Check uniqueness on the lookup side, preserve the request IDs and inspect unmatched entities. Pandas merge validation can enforce those assumptions rather than silently accepting duplicate matches.

Historical features and fitted group statistics use different boundaries

A historical order count is recomputed at each row's cutoff from eligible observed events. A fitted reference statistic, such as a store's average parcel weight used as a lookup, is estimated on the training partition and then held fixed for that validation fold. State which behavior the feature represents.

A group average of the target is more sensitive. If a row's own outcome enters its feature, the model can partially read its answer. Use appropriate cross-fitting for training rows and fit the final lookup from training labels for later predictions. For temporal tasks, labels must also have become available by the relevant cutoff. The target-encoding chapter handles that case in detail.

Implement the eligibility rule directly first

The reference implementation scans the small event table for each request. Timestamps are written as minutes relative to a fixed UTC origin to keep the example compact; yesterday's times are negative. The three comparisons encode the window and availability rule exactly.

import pandas as pd

# Minutes relative to 14 September 2026, 00:00 UTC, for this small example.
raw = [
    ('E0', 'A', -870, -869, 10),
    ('E1', 'A', -720, -719, 20),
    ('E2', 'A',  540,  541, 40),
    ('E3', 'A',  590,  610, 60),
    ('E4', 'A',  600,  600, 100),
    ('E5', 'A',  660,  661, 80),
    ('E6', 'B',  480,  481, 999),
]
events = pd.DataFrame(raw, columns=['event_id', 'customer', 'event_min',
                                  'available_min', 'amount'])
# Assume validated records: unique events, known times, finite amounts.
assert events.event_id.is_unique

def aggregate_asof(events, customer, cutoff):
    eligible = events.loc[
        events.customer.eq(customer)
        & events.event_min.ge(cutoff - 1440)
        & events.event_min.lt(cutoff)
        & events.available_min.le(cutoff)
    ]
    n = len(eligible)
    return {
        'orders_24h': n,
        'spend_24h': float(eligible.amount.sum()),
        'mean_amount_24h': float(eligible.amount.mean()) if n else None,
    }

queries = pd.DataFrame([
    ('P1', 'A', 600), ('P2', 'A', 615),
    ('P3', 'A', 675), ('P4', 'A', 735), ('P5', 'C', 600),
], columns=['prediction_id', 'customer', 'cutoff_min'])
features = pd.DataFrame([
    {'prediction_id': q.prediction_id,
     **aggregate_asof(events, q.customer, q.cutoff_min)}
    for q in queries.itertuples(index=False)
])
result = queries.merge(features, on='prediction_id', how='left',
                       validate='one_to_one')
print(result[['prediction_id', 'orders_24h', 'spend_24h', 'mean_amount_24h']])

The same rule in C++

Keep the mean optional until at least one eligible amount exists. This example uses already validated, deduplicated records; it demonstrates membership and accumulation rather than a storage system.

#include <iostream>
#include <optional>
#include <string>
#include <vector>

struct Event {
    std::string customer;
    int occurred, available;  // Relative UTC minutes in this example.
    double amount;
};
struct Features {
    int count = 0;
    double sum = 0;
    std::optional<double> mean;
};

// Assumes validated, deduplicated events and bounded minute timestamps.
Features aggregate_asof(const std::vector<Event>& events,
                        const std::string& customer, int cutoff) {
    Features result;
    for (const auto& event : events) {
        if (event.customer != customer || event.occurred < cutoff - 1440
            || event.occurred >= cutoff || event.available > cutoff) continue;
        ++result.count;
        result.sum += event.amount;
    }
    if (result.count) result.mean = result.sum / result.count;
    return result;
}

int main() {
    std::vector<Event> events = {
        {"A", -870, -869, 10}, {"A", -720, -719, 20},
        {"A", 540, 541, 40}, {"A", 590, 610, 60},
        {"A", 600, 600, 100}, {"A", 660, 661, 80},
        {"B", 480, 481, 999},
    };
    for (int cutoff : {600, 615, 675, 735}) {
        auto f = aggregate_asof(events, "A", cutoff);
        std::cout << f.count << ' ' << f.sum << ' ' << *f.mean << '\n';
    }
    // New customer: count=0, sum=0, mean has no value.
    auto cold = aggregate_asof(events, "C", 600);
    std::cout << cold.count << ' ' << cold.sum << ' ' << cold.mean.has_value() << '\n';
}

For many requests, repeatedly scanning every event becomes expensive. Use indexed temporal joins, precomputed snapshots or a carefully maintained rolling state. Preserve the reference routine as a correctness check. Sorting by event time and shifting a rolling statistic is not sufficient by itself when records arrive late or are corrected later.

Choose windows that express plausible behavior—recent activity versus a longer-term baseline—and validate them as model choices. Keep denominators, coverage and freshness interpretable alongside the summary. More aggregate columns are useful only when they add reliable information at the actual prediction time.

References and further reading

• Feast: point-in-time joins — historical retrieval, per-row time boundaries and availability filtering for backfilled values.
• Featuretools: handling time — cutoff times, training windows and columns that become known at different times.
• Pandas: DataFrame.merge — one-to-one and many-to-one join validation.
• Scikit-learn: TargetEncoder — cross-fitting when a group statistic uses outcome labels.