Feature Stores at Scale: Uber, Netflix, Airbnb

Why Feature Stores at Scale?

When an organization runs hundreds of ML models in production, feature engineering becomes a bottleneck. Data scientists spend 60–80% of their time wrangling data, and without a centralized feature store, teams duplicate effort, introduce inconsistencies, and struggle with training/serving skew.

At companies like Uber, Netflix, and Airbnb, the scale is staggering:

• Billions of feature values computed and stored daily
• Millisecond latency requirements for real-time serving
• Hundreds of ML models consuming shared features
• Thousands of feature definitions maintained by dozens of teams
• Petabytes of historical data for training and backfill

These constraints forced each company to build custom feature store infrastructure. Their architectures evolved through years of production experience, and the patterns they discovered are now shaping the broader MLOps ecosystem.

Uber's Michelangelo

Uber's Michelangelo platform was one of the earliest large-scale ML platforms, and its feature store component—Palette—set many of the patterns the industry now follows. Built to serve features for ETA prediction, fraud detection, driver matching, and dynamic pricing, Palette had to handle extreme scale and latency requirements from the start.

Architecture

Palette uses a dual-store architecture. The offline store is built on Hive tables stored in HDFS, providing petabyte-scale storage for historical features used in training. The online store uses Cassandra, optimized for high-throughput key-value lookups. A materialization pipeline, running on Spark, moves features from offline to online on a schedule.

Real-Time Features

For features that require sub-minute freshness—like "number of ride requests in the last 5 minutes in this geo-cell"—Uber built a streaming pipeline on Apache Flink. Events from Kafka are aggregated in real-time and written directly to the online store, bypassing the batch path entirely.

Key Innovations

• DSL for feature definitions: A domain-specific language lets data scientists define features declaratively, which the platform compiles into Spark or Flink jobs.
• Feature versioning: Every feature definition is versioned, and training datasets are logged with the exact feature versions used—enabling reproducibility.
• Geo-partitioned features: Features are partitioned by H3 geo-cells, enabling spatial queries and location-aware aggregations at scale.

Here's an example of how Uber-style DSL feature definitions translate to platform jobs:

# Uber-style declarative feature definition (DSL)
from platform.features import Feature, Entity, Window

ride_entity = Entity(name="rider", keys=["rider_id"])

# Batch feature — compiled to Spark job
rider_lifetime_trips = Feature(
    name="rider_lifetime_trips",
    entity=ride_entity,
    source="trips_table",
    aggregation="count",
    window=Window(start="entity_creation", end="now")
)

# Streaming feature — compiled to Flink job
recent_requests = Feature(
    name="geo_ride_requests_5m",
    entity=ride_entity,
    source="ride_request_stream",
    aggregation="count",
    window=Window(duration="5m", partition_by="h3_cell"),
    serving="online_only"
)

Netflix's Feature Store

Netflix's approach to feature management is deeply integrated with their broader data platform, Cosmos. Rather than building a standalone feature store, Netflix embedded feature management into their data mesh—a decentralized architecture where domain teams own their data pipelines and feature definitions.

Cosmos Platform

Cosmos is Netflix's compute platform for data and ML workflows. It orchestrates Spark jobs, manages data dependencies, and provides a unified interface for batch and streaming computation. Feature pipelines are just another type of Cosmos workflow.

Data Mesh Philosophy

Netflix treats features as data products. Each team (recommendations, search, content, studio) owns their feature pipelines end-to-end. A central catalog provides discoverability, but there's no single team that "owns the feature store." This decentralized model scales better organizationally but requires strong conventions and tooling.

Feature Management

Netflix's feature store provides:

• Feature catalog: A searchable registry of all features across the organization, with metadata, lineage, and quality metrics.
• Temporal correctness: Point-in-time join infrastructure that prevents data leakage in training datasets.
• Feature monitoring: Automated drift detection and data quality checks that alert teams when feature distributions shift.
• Shared compute: Common aggregation patterns (count, sum, avg over time windows) are provided as reusable building blocks.

Netflix's Cosmos-style feature pipeline definition follows a data-product pattern:

# Netflix Cosmos-style feature pipeline (data product)
from cosmos.pipelines import FeaturePipeline, SparkJob
from cosmos.catalog import register_feature

class UserEngagementFeatures(FeaturePipeline):
    owner = "recommendations-team"
    schedule = "@hourly"

    def compute(self, spark):
        viewing = spark.read.table("viewing_history")
        return viewing.groupBy("user_id").agg(
            count("title_id").alias("titles_watched_24h"),
            avg("watch_pct").alias("avg_completion_rate"),
        )

# Register as discoverable data product
register_feature(
    pipeline=UserEngagementFeatures,
    tags=["engagement", "recommendations"],
    sla_freshness="2h"
)

Airbnb's Zipline

Airbnb's feature store, Zipline, was built with a clear north star: make it trivially easy for data scientists to define features and have the platform handle all the complexity of backfill, serving, and consistency. Zipline powers features for search ranking, pricing, fraud detection, and host recommendations.

Feature Framework

Zipline provides a Python-based feature definition framework. Data scientists write feature transformations as simple Python functions annotated with metadata (entity, data source, aggregation window). The framework handles compilation to Spark, scheduling, and materialization.

Backfill

One of Zipline's most celebrated capabilities is automatic backfill. When a new feature is defined, Zipline can automatically compute its values over historical data—months or years of events—so that data scientists can immediately use the feature for training without waiting for data to accumulate.

Time-Travel

Zipline implements rigorous time-travel semantics. For any entity at any point in time, you can retrieve the exact feature values that would have been available at that moment. This eliminates a class of subtle data leakage bugs that plague feature engineering at scale.

Key Design Decisions

• Immutable feature logs: Raw feature values are stored as immutable event logs, enabling recomputation when definitions change.
• Consistency guarantees: The same feature definition produces identical values in batch training and online serving—no separate code paths.
• Self-serve onboarding: A new feature can go from definition to production in hours, not weeks.

Airbnb's Zipline-style feature definition with automatic backfill:

# Airbnb Zipline-style feature definition
from zipline.features import Feature, WindowAgg, Source

class HostResponseFeatures:
    entity = "host_id"
    source = Source(table="messages", event_ts="sent_at")

    # Auto-backfilled over 2 years of history
    avg_response_time_7d = Feature(
        agg=WindowAgg.AVG("response_seconds"),
        window="7d",
        backfill=True,
        backfill_start="2022-01-01"
    )

    response_rate_30d = Feature(
        agg=WindowAgg.RATIO("responded", "received"),
        window="30d",
        backfill=True,
        backfill_start="2022-01-01"
    )

# Time-travel: get features as of a specific timestamp
from zipline.retrieval import get_features_at
features = get_features_at(
    entity_key={"host_id": "H12345"},
    feature_class=HostResponseFeatures,
    as_of="2024-03-15T10:00:00Z"
)

Common Architectural Patterns

Despite different technology choices, Uber, Netflix, and Airbnb converged on remarkably similar architectural patterns. These shared patterns represent hard-won wisdom from years of operating feature stores in production.

Unique Approaches

The differences are as instructive as the similarities:

• Uber's custom DSL enables platform-level optimization but requires engineers to learn a new language.
• Netflix's data mesh scales organizationally but demands strong conventions to prevent fragmentation.
• Airbnb's auto-backfill dramatically accelerates adoption but requires significant compute infrastructure.

Key Lessons Learned

Across all three companies, several lessons emerge repeatedly. These aren't theoretical—they're battle-tested insights from operating feature stores that serve billions of predictions daily.

1. Start Simple, Then Evolve

Every company started with batch features and a simple offline store. Streaming, real-time computation, and sophisticated governance were added incrementally as needs arose. Don't over-engineer your first feature store—get features flowing and iterate.

2. Invest Heavily in Metadata

All three companies emphasize that the metadata layer—feature definitions, lineage, ownership, quality metrics—delivers more long-term value than the storage or compute layers. Features are only useful if people can find them, trust them, and understand their provenance.

3. Automate Backfill from Day One

Airbnb's experience shows that backfill capability is the single biggest driver of adoption. If a data scientist defines a new feature and has to wait 30 days for data to accumulate, they'll bypass the feature store entirely.

4. Monitoring Is Not Optional

Feature quality degrades silently. A schema change in an upstream table, a bug in a transformation, or a shift in data distribution can corrupt features without triggering any errors. All three companies learned (often painfully) that feature monitoring must be built into the platform from the start.

5. Organizational Design Matters

The success of a feature store depends as much on organizational decisions as technical ones. Uber chose a centralized platform team, Netflix chose a decentralized data mesh, and Airbnb chose a hybrid. The right choice depends on your org's size, culture, and data maturity.