Level Up Your Kafka Skills in Just 5 Days | Join Season of Streaming On-Demand

Presentation

Kafka Streams Rebalances and Assignments: The Whole Story

« Kafka Summit London 2023

Kafka Streams makes it very easy to run a distributed stream processing system: you just start multiple processes with the same ""application id"", point them at the same Kafka cluster, and they will form a distributed system on their own. Making a distributed system so easy to start up has an Achilles' heel: there's no natural learning curve that forces you to learn how to operate the system in order to stand it up.

Of all the pitfalls this situation creates, by far the most complex and user confusing is the topic of rebalancing and task assignment. Kafka Streams partition and task assignment is especially complex because of advanced features like ""smooth scale-out"" that uses warm-up tasks and so-called probing rebalances. This talk aims to completely demystify the system from an operational perspective. You will learn what is really happening across Kafka and Kafka Streams, how to interpret the logs and metrics, and how to adjust the configs to achieve your desired outcomes.

Related Links

How Confluent Completes Apache Kafka eBook

Leverage a cloud-native service 10x better than Apache Kafka

Confluent Developer Center

Spend less on Kafka with Confluent, come see how