Infinite retention for Apache Kafka^® with Confluent

In traditional data architectures, storage systems focus on accumulating past records while messaging services are waiting to process future events. On top of this data layer, there may be hundreds of in-house systems, SaaS applications, or microservices both generating and consuming data. Data integrity and correctness must be maintained across all these systems, which is a huge operational burden and business risk as organizations and apps must make quick decisions on what they believe is the most accurate and up to date version of the data while retaining what is needed for regulatory compliance. Confluent's event streaming solutions allow you to:

• With no limits on storage, retain all historical and current events in Kafka as long as you want
  • Set retention at the topic level to be as long as you want without scaling additional compute resources to handle the storage
  • Reduce tech debt and downtime risk with a central source of truth for all events across your entire ecosystem
• Democratize access to your data with all events in one place and eliminate the need for complex data engineering
  • Maintain data compatibility with Schema Registry and enforce data integrity with Schema Validation to build an infinite and immutable central source of truth for all streaming events
  • Keep the flexibility to use your programming language of choice and transform infinite amounts of data in Kafka using clients such as C, .NET, Java, GO, Python and more
• Reduce business risk by setting retention policies that help maintain regulatory compliance and providing a persistent log of all events for compliance audits

Provisioning massive Kafka clusters and retaining the right amount of data in the cloud with Apache Kafka is complex. Setting it up properly to handle GBps throughputs and petabytes of data can turn into months of work that push delivery timelines delaying business go-to-market. Similarly, businesses that don't need GBps scale or petabytes of storage at the beginning but face unexpected spikes or storage needs risk causing downtime or degrading cluster performance, which can lead to customer frustration. Confluent Cloud allows you to:

• Never worry about Kafka storage again in Confluent Cloud with elastically scalable storage that automatically grows with your traffic
  • Avoid downtime and data loss by auto-scaling storage based on retention policies and traffic
• Only pay for storage used with Confluent Cloud instead of paying for over provisioned storage volumes attached to Kafka brokers
  • Only pay for retained data, not pre-provisioned storage capacity
  • Decouple storage and compute to avoid unnecessary compute costs for high storage use cases

Storing historical data on self-managed Kafka clusters quickly increases your infrastructure costs because you have to over provision storage to prevent downtime and deploy larger clusters to avoid hitting storage limitations on brokers. It becomes cost-prohibitive to retain all the data you need in Kafka, causing organizations to glue together disjointed architectures that end up costing more money and siphoning off more engineering resources. Confluent Platform allows you to:

• Send long term storage to cost-efficient object storage, such as S3 and GCS, using Tiered Storage in Confluent Platform
  • Expand cluster storage in Confluent Platform without any additional resource provisioning when you offload data to 3rd party object storage
• Reduce partition rebalancing time for self-managed clusters from hours to seconds using Tiered Storage to minimize the size of your partitions on brokers