Optimizing for Low Latency and High Throughput

A Sticky Partitioner was introduced in Kafka 2.4 to improve record batching, but it caused performance problems by directing more data to slower brokers. To fix this problem, a new Sticky Partitioner was developed to direct data to faster brokers to provide better latency and throughput for unkeyed messages, as described here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-794%3A+Strictly+Uniform+Sticky+Partitioner.

In this talk I'll cover a simple, but effective algorithm for auto-tuning effective batch size for low latency and high throughput, adaptive partitioning logic to direct more data to faster brokers, and go through benchmark results that illustrate effectiveness of the new Sticky Partitioner.