10 Things We Learned Re-imagining Kafka for the Cloud

In 2017, Confluent launched a cloud product. For the most part, it was Apache Kafka® run on a Cloud Service Provider by a team of SREs. It wasn’t great. We got caught thinking pre-cloud, but cloud environments change everything. We took a step back and questioned the entire system. We needed to re-imagine what a streaming platform could be.

Fast-forward 3M+ people hours later, and Confluent has re-engineered Kafka into a robust, performant, and serviceless streaming service. In this talk, I’ll walk through the top 10 lessons learned re-imagining Kafka for the Cloud, from storage to data balancing, to scaling and security.