Lessons Learned Scaling Stateful Kafka Streams Topologies

Kafka Streams is a powerful engine to do stream processing, and its stateful operations have allowed us to implement event-driven architectures in a simple, efficient and productive way. Our use case is about real estate listings websites, and at relatively low volumes of data (few millions) everything worked out of the box. However, when we started scaling things got a bit more difficult: High latency on every rolling-update, topologies eternally in rebalance, write stalls, excessive AWS bills and even losing data. I will explain a bunch of actions we have done that helped us scale our topologies to process hundreds of millions of listings: Use kubernetes StatefulSets, tune RocksDB configurations, use Horizontal Pod Scaling wisely, activate consumer Rack Awareness, and more.