Exactly-Once Stream Processing at Scale in Uber

Uber successfully leverages multiple technologies, such as Kafka, Flink, Pinot and others to accommodate a wide spectrum of real time use cases such as data transformation, processing,alerting, fraud & threat detection, analytics, archival etc.

Many business use cases have been steadily emerging at Uber with a need for Exactly Once semantics. Tracking Ads spend is one such use case that is particularly important.

In this presentation, Si and Roshan will shed light on how unified exactly-once guarantees are supported within Uber’s Data platform, emphasizing the integration between multiple streaming technologies to address the specific needs of exactly-once use cases. We will also provide an overview of the thought process underpinning the platform's recent enhancements and prospective roadmap.

Topics include:

1. A brief overview of Uber's real-time streaming data infrastructure, covering Kafka, Flink, and Pinot.
2. Enhancements to the Kafka, Flink platforms to enable and simplify building end-to-end exactly-once pipelines at scale.
3. Unique problems we tackled, open challenges, as well as our future roadmap.