Trials, Tribulations, and Triumphs: Migrating from Self-Managed Kafka to Managed Kafka in the Cloud

Compare the Market has long used Kafka as part of its event data streaming architecture. Whilst Kafka proved a reliable and scalable solution, hosting it in AWS and managing it ourselves came with its own challenges. These challenges included administration of the platform and security, not to mention handling incidents and needing highly skilled talent in the teams to do all of this - and were all contributing factors in driving the decision to migrate.

After much discovery, we landed on ​​Confluent Cloud as our provider of choice to replace our existing Self-Managed Kafka infrastructure. Attend this session to learn about how we went about the migration and issues faced and how this will power our next generation data platform. We will discuss how we overcame the following challenges and more:

• Identifying a method of replicating topics between platforms
• Reproducing our existing infrastructure as code with the Confluent Platform and Terraform
• Coordinating the migration of 100+ consumers and producers, used across the business