Crossing the Streams: the New Streaming Foreign-Key Join Feature in Kafka Streams

Have you ever wished that KTable joins worked like SQL joins? Well, now they do! Foreign-key, many:one, joins were added to Apache Kafka in 2.4. This talk is a deep dive into the surprisingly complex implementation required to compute these joins correctly. Building on that understanding, we'll discuss how you can expect Streams to behave when you use the feature, including how to test it, and thoughts on optimization. Finally, we will take Bazaarvoice as a case study. They are in process on migrating from their high-scale in-house stream processing platform to one based on Apache Kafka and Kafka Streams. I'll share the way that they implemented foreign-key joins on Kafka 2.3, and how much simpler it is with native support. Plus, we will also share key operational insights from their experience.