Chill, Distill, No Overkill: Best Practices to Stress Test Kafka

So, you have built/inherited/discovered one of your many Kafka clusters. How now do you know that it is good enough to sustain and grow your applications? Do you stress test it as a data store, a messaging system, as middleware, or like a REST API? Or are you in production and worried about the next unprecedented surge? Find out from those who have asked and answered before.

Repeatable, and recordable stress testing for Kafka is a challenge for novices and some legends. Real supplies like storage, compute, network, threads etc. do not naturally map to demands of messages, bytes, and milliseconds. In the session, we will cover ways to:

• Define parameters and variables before beginning

• Accommodate for changing conditions - brokers, applications, config, network

• Overlap infrastructure, test design, latency, and throughput

• Meet cost, service level agreements, and multi-tenancy needs while testing

• Do it all without entirely relying on estimation, and extrapolation

We will also discuss common and innovative practices observed in the industry to meet this challenge. At the end of the session, you would walk away with the knowledge needed to set up a repeatable stress test suite without stress.