Evolution of Real-time User Engagement Event Consumption at Pinterest

We will discuss how we at Pinterest transformed real time user engagement event consumption.

Every day, we log hundreds of billions of user engagement events across different domains to a few common Kafka topics which are consumed by hundreds of real time applications. These real time applications were built upon diverged frameworks (e.g. Spark Streaming, Storm, Flink, and internally developed frameworks using Kafka Consumer API) without standardization on processing logics. It led to repeated processing of similar logic, multiple codebases to maintain, low data quality, and inconsistency with offline datasets. These negatively impact scalability, reliability, efficiency and data accuracy of these applications and eventually affect the real-time content recommendation quality and user experience.

To address these challenges, we unified the way of consuming events in our real time applications by consolidating the compute engines to Flink, splitting events in those common topics by engagement types, generating cleansed events with standardized processing to align on business concepts. Throughout these efforts, we achieved multi-million dollar infrastructure savings and double-digit engagement gain after applications adopted those cleansed events.

Moving forward, we are implementing frameworks for better tracking and governing the Kafka events and real time use cases.