Evolution of Streaming Pipeline at Lyft

Lyft is a ride-sharing company which is a two-sided marketplace; balancing supply and demand using various levers (passenger pricing, driver incentive etc.) to maintain an efficient system. Lyft has built a real-time optimization platform that helps to build the product faster. This complex system makes real-time decisions using various data sources; machine learning models; and a streaming infrastructure for low latency, reliability and scalability. This infrastructure consumes a massive number of events from different sources to make real-time product decisions.

Rakesh discusses how Lyft organically evolved and scaled the streaming platform that provides a consistent view of the marketplace to aid an individual team independently run their optimization. The platform offers online and offline feature access that helps teams to back test their model in the future.

Each iteration provided better scale, unlocking different capabilities of the marketplace and new avenues for growth. To accomplish these, the team has set the roadmap for next-generation streaming platform and developing smarter tools.

Topics include:

• Describing Lyft’s streaming platform for dynamic decision-making

• Where Kafka fits in streaming tech stack and how it helped us to scale better

• Productionizing the very first pipeline

• Tools that simplified pipeline creation development environment that is friendly to Data Science people

• Lesson learned