Level Up Your Kafka Skills in Just 5 Days | Join Season of Streaming On-Demand
As soon as you run Kafka Connect with more than one worker you will likely encounter rebalancing. In the past Connect’s “stop-the-world” rebalancing approach meant that you needed to avoid it at all costs. However we’ve come a long way from the days when starting a new connector affected all your existing workloads.
In this session we will look at how the rebalancing strategies in Connect have evolved over time. We’ll explain the different strategies that are in place now and, crucially, the effect they have on your connectors and the flow of data through your pipelines. This session will help you understand not only the impact of adding and removing workers, but also explain how to configure Connect to prevent unnecessary data processing pauses during updates.
Finally we will look at how rebalances affect Connect clusters running on Kubernetes. We’ll look at the types of resources that are suitable for Connect workers and what rolling update strategy you need to use to work with the rebalance strategy, rather than against it.