Availability-first Kafka Streams at Walmart Scale

At Walmart, we offer a free delivery subscription service for customers. To prevent it from being misused, we run a fraud detection model on every transaction, and the model comes with very tight SLAs. So, we had to optimize for high availability and low serving latency in the Kafka streams cluster. In this talk, you’ll learn about our platform architecture and how we meet our SLA requirements. It includes discussing the tools we used for analyzing and tuning performance, and details of a custom client library that we wrote for Kafka Streams to query the precise machine on which state resides. We’ll also discuss improvements that we made to Apache Kafka, including:
• Serving reads from standbys (KIP-535)
• Serving reads while rebalancing (KIP-535)
• Reading from a store of a specific partition (KIP-562)
All of these enable us to deliver Kafka Streams applications that are focused on availability and throughput. Screen reader support enabled.