The Age of the Clusters: Offering Kafka as a Service in Your Organisation

Every team in your organisation wants to leverage the latest and greatest technologies, Kafka is of course at the top of the list but should you run one shared cluster or one per team? Should you build and deploy it yourself or leverage a managed service? This talk will dive into the journey you must take in order to reach your ultimate goal making Kafka the commodity all your development teams run on.

Building a shared platform service is not like building and shipping an application in your favorite shiny modern programming language. Kafka is stateful and evolves with the use cases built upon it. Building a cloud native Kafka-as-a-Service offering in house is hard, it requires a clear vision, sound automation and committed engineering team with a lot of patience.

Confluent Kafka for Kubernetes is a cloud native way of deploying Kafka, at scale, in a self managed way. But how does this help? Combining this technology with GitOps enables developers to precisely describe the desired state of their clusters in each environment.

I will be showcasing the key tools and techniques required to make Kafka a commodity and automate operations through Git in a repeatable, secure and controlled manner.