Announcing Confluent Cloud: Apache Kafka® as a Service

Today, I’m really excited to announce Confluent Cloud^TM, Apache Kafka^® as a Service: the simplest, fastest, most robust and cost effective way to run Apache Kafka in the public cloud. Confluent Cloud is a fully managed streaming data service that makes running Kafka almost effortless, whether you are running a few applications or as a central nervous system that acts as the source-of-truth for all data in a company, serving trillions of messages a day and hosting several terabytes of data.

This is a significant step forward for Kafka users everywhere. Today, thousands of companies are running the core of their business on Kafka. As the original creators of Apache Kafka, we have had the pleasure of interacting with that broad Kafka user base, building a thriving open-source community around it and helping Confluent’s customers transform their data infrastructure through streaming data. From that experience, it is clear to me that developers love Kafka; we want to use Kafka to serve as the company’s central nervous system, a streaming backbone for all our microservices, a commit log to serve database changelogs, a basis for stream processing, but we don’t want to operate Kafka. Why? Because running distributed systems in the cloud on your own is really, really hard. As developers, we expect distributed systems to just be available as managed services so we can focus on building applications; not manage infrastructure and data services. It is my observation that an increasing number of companies are either starting entirely in or making a shift to the public cloud and want to use Kafka as a hosted service in the cloud. We believe that all developers should be able to spin up a Kafka-based streaming platform as quickly as they spin up compute.

At Confluent, we are committed to our mission of building a streaming platform and putting it at the heart of every company. We are proud that, as a company, we have made major strides in achieving our stated objective and Confluent Cloud is a natural step forward in our 7-year long journey so far.

Here’s the vision we have for Confluent Cloud:

Your favorite Kafka, made simple

Confluent Cloud is everything about Apache Kafka that you’ve come to know and love, made simple. All the capabilities that make Kafka the leading streaming technology, from its legendary performance to its unique retention and replay capabilities, are included. You pick throughput, retention, and replication tiers and click go. And that’s it. You will get a secure Kafka cluster sized and provisioned appropriately for your application’s needs, in a matter of seconds. There are no brokers to setup, configure, or manage, no upgrades or backups to schedule, no need to set up monitoring or look for security vulnerabilities. If a broker goes down at night, Confluent Cloud takes care of it for you. And if something cannot be fixed automatically, our 24/7 response team is paged, not you. When you need to increase the retention of your topics, run on bigger hardware, or add more partitions, it’s a few clicks in the UI, and Confluent Cloud takes care of the rest.

Not just Kafka, but Kafka’s entire open-source ecosystem

Confluent Cloud offers the same open-source Apache Kafka APIs and supports the open-source Kafka protocol. This means, you not only get Kafka as a service, but also access to — and freedom to adopt — anything that’s compatible with Kafka from the large and thriving open-source ecosystem around it. Over the last two and half years, Confluent has created several tools that developers need to succeed with Kafka: clients in various languages, REST proxy, Schema Registry, and many open-source Kafka connectors. All of this will be available in Confluent Cloud, the only hosted Kafka offering that includes Confluent’s entire streaming platform.

Robust and performant; years of operational expertise as a service

Confluent Cloud is made by the team that originally created Apache Kafka. It builds on 7+ years of operational expertise that our team has gathered from running Kafka as a service for thousands of developers and hundreds of applications at LinkedIn to supporting Confluent’s customers doing the same. From all that experience so far, I’ve learned that running distributed systems is hard and operations is the hardest part of running distributed systems effectively. Operational excellence comes with time and experience. We’ve dealt with and fixed countless creative failure scenarios, upgraded Kafka numerous times while never taking it offline, set the right alerts, developed the right runbooks, and mastered the dark art of observability and monitoring. This operational experience is now available to you through Confluent Cloud.

No lock-in; freedom to deploy Kafka in any public cloud

As an industry, we have moved away from proprietary systems that store our data and lock us in. In the last decade, that has led to the shift away from proprietary systems and widespread adoption of open-source data infrastructure. This will play out the same way as we traverse of the arc of the shift to the public cloud as well. Public cloud services that require developers to program to proprietary APIs will not succeed. You need the freedom to take your applications and your data and move to any other cloud provider or use more than one cloud for different applications.

We want to enable developers to move faster, not slow them down with the worry of vendor lock-in and the burden of an arduous migration process. In keeping with that vision, we will make Confluent Cloud available on all the major public cloud providers, starting with AWS and later, Azure and Google Cloud. With Confluent Cloud, you can program your applications to the same open-source Apache Kafka APIs and migrate your applications to another public cloud without having to rewrite them. This is a major advantage of using Confluent Cloud and one that differentiates it from streaming data services native to the public cloud providers.

Early access for the early birds

As with some of our existing users, I’m sure you can’t wait to get up and running on Confluent Cloud. As part of launching the product, we are making it available to a handful of users first before releasing it as a generally available service soon. Sign up here to be part of this early access program.

Sign up for the early access

Confluent: Kafka everywhere; on-prem or in the cloud

To say that I’m excited about the arrival of Confluent Cloud would be an understatement. This is a major leap forward in increasing the reach of Kafka for cloud-first developers as well as enterprises transitioning to the cloud. It is a big step towards realizing our vision at Confluent — to build a streaming platform, and put it at the heart of every company — in every way, be it on-prem, or in the cloud.

Want to learn more? Join us for an online talk featuring Senior Director of Engineering Rob Benson. In this talk, Rob will walk through an overview of Confluent Cloud and discuss how to build robust, portable and lock-in free streaming applications. Register now!