The original creators of Apache Kafka once again set the foundation for the future of event streaming; re-engineered platform delivers new levels of serverless scale and cost efficiencies
Under Project Metamorphosis, elastic scaling is the first and foundational step in establishing the next generation of event streaming
Mountain View, Calif. – May 6, 2020 – Confluent, Inc., the event streaming platform pioneer, today announced the launch of elastic scaling for Apache Kafka® in Confluent Cloud. Following the company’s $250 million series e round, Confluent is ushering in the future of event streaming with Project Metamorphosis. Today, the company unveils the first of this series of new innovations that accelerate how companies can harness the full power of event streaming at any scale. The Elastic release sets new levels of elasticity that were never before possible in event streaming without extensive Kafka expertise or large teams of developers working around the clock. Companies can now instantaneously harness the power of event streaming to meet the real-time urgency and unprecedented levels of uncertainty for businesses today.
These past few months have completely reshaped how businesses interact with customers. Applications have become many companies’ sole touchpoints, making business success even more reliant on exceptional digital experiences. Adapting and scaling these applications to quickly meet customer demand is increasingly important, but also increasingly difficult as most applications are built on rigid data architectures that span legacy technology and modern cloud environments. In addition to technical challenges, businesses also end up either wastefully overspending on resources they never or rarely use or get caught off guard by unexpected traffic surges, causing poor application performance and downtime. The only way to build an application that can scale on demand and operate cost efficiently is with elastic, cloud-based architecture.
“Elasticity is a fundamental property of cloud data systems and our first step in Project Metamorphosis is bringing elastic scaling to Kafka and it’s ecosystem in Confluent Cloud,” said Jay Kreps, co-founder and CEO, Confluent. “This is particularly important in uncertain times like this where we see many of our customers needing to suddenly scale up the digital side of their business as that becomes their primary channel of serving their customers.”
Kafka has risen as the de facto standard for event streaming with thousands of enterprises using it, including more than half of the Fortune 100. Although it has widespread adoption, many companies are not able to realize their full potential through real-time event streaming because of the level of Kafka expertise and developer power needed. The original creators of Kafka founded Confluent to eliminate these roadblocks.
The Elastic release unveils new levels of elasticity that were never before possible in event streaming through innovations in Self-Balancing Clusters planned for a future Confluent Platform release; as well as on-demand cluster provisioning and expansion and a usage-based billing model for all of Confluent Cloud. As a result, organizations are empowered to use event streaming no matter the scale of the use case, budget of the project, or experience with Kafka.
Remove the Operational Burden of Balancing Clusters
When organizations make event streaming a central platform for all their data, use cases can quickly grow to manage thousands of topics and produce trillions of messages a day. As new topics are added and old brokers are retired, it requires teams to reassign partitions and balance workloads to avoid overloading clusters and causing failures. Reshuffling massive loads of data can be daunting and tedious, especially as these platforms are connected to critical applications. To make this dramatically easier, Confluent plans to release Self-Balancing Clusters in Confluent Platform, which automates resource workload balancing, provides failure detection and self-healing, and allows customers to easily add and decommission brokers. With Confluent Platform, companies self-managing Kafka can focus more resources on building exceptional real-time experiences with less risk and manual effort.
Self-Balancing Clusters are currently planned for a future Confluent Platform release.
Elastically Scale Apache Kafka On Demand
Scaling data architecture on demand is essential for handling unexpected spikes in traffic and avoiding application lag or downtime. However, most distributed computing architectures, including open source Apache Kafka, are incredibly challenging to scale because of the amount of custom coding and deep level of technical expertise needed to connect every data source. A new innovation in Confluent Cloud solves this issue by enabling anyone to instantly provision and expand Dedicated clusters with a few clicks. This builds on the elasticity already available in Basic and Standard clusters that instantly scale up and down between 0-100 MBps. Serverless properties like these, across the entire Confluent Cloud solution, enable companies to grow and shrink production-level use cases on demand without having to issue support tickets or go through the complex process of manual resizing, saving significant amounts of time and resources.
Self-serve provisioning is available today and expansion for Dedicated clusters will be available later in May.
Operate Efficiently by Paying Only for Data Streamed
Because organizations want to be prepared for high-traffic moments and future growth, they often end up paying for peak provisioned capacity, which can be up to 10x higher than what’s actually consumed. With greater scrutiny over costs, the pay-as-you-go model associated with born-in-the-cloud data systems has become the new standard. Confluent Cloud was engineered with pay-for-what-you-use in mind and Confluent is extending this model to every part of its cloud service. Customers can now commit to spend a certain amount across the entire cloud portfolio, including fully managed stream processing with Confluent Cloud KSQL and Confluent’s fully managed Kafka connectors, and pay only for the resources they use. As a result, organizations can operate event streaming with greater cost efficiencies for any budget.
Usage-based billing is now available across Confluent’s entire cloud portfolio.
Established Innovations in Elastic Scaling
The innovations announced today join a host of features already available across Confluent Cloud, Confluent Platform, and open source Kafka, and are designed to make event streaming truly elastic. These innovations include:
Confluent Cloud
Confluent Platform
Apache Kafka
Currently, Apache Kafka uses Apache ZooKeeper™ to store its metadata such as the location of partitions and the configuration of topics are stored outside of Kafka itself, in a separate ZooKeeper cluster. However, this concept of external metadata management forces system administrators to manage two systems rather than one, inevitably leading to different configuration systems, management interfaces, and security settings. This leads to an unnecessarily steep learning curve, inefficiencies, and is the biggest bottleneck in Kafka’s scalability. To make event streaming more scalable for everyone, the Apache Kafka community has made tremendous progress toward the removal of ZooKeeper (KIP-500). Once complete, Kafka will be able to support millions of partitions with better resilience and faster failover times.
Project Metamorphosis
Confluent will continue to make new product announcements as a part of Project Metamorphosis through the rest of the year. On the first Wednesday of every month, Confluent will unveil a set of capabilities that address the major technical challenges organizations face when putting event streaming at the heart of their business, laying the foundation to make pervasive use of event streaming possible for any organization.
Additional Resources
About Confluent
Confluent, founded by the original creators of Apache Kafka®, pioneered the enterprise-ready event streaming platform. With Confluent, organizations benefit from the first event streaming platform built for the enterprise with the ease of use, scalability, security and flexibility required by the most discerning global companies to run their business in real time. Companies leading their respective industries have realized success with this new platform paradigm to transform their architectures to streaming from batch processing, spanning on-premises and multi-cloud environments. Confluent is headquartered in Mountain View and London, with offices globally. To learn more, please visit www.confluent.io. Download Confluent Platform and Confluent Cloud at www.confluent.io/download.
Confluent and associated marks are trademarks or registered trademarks of Confluent, Inc.
Apache® and Apache Kafka® are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. No endorsement by the Apache Software Foundation is implied by the use of these marks. All other trademarks are the property of their respective owners.