New in Confluent Cloud: Unleashing Cost-Effective Streaming for Any Workload

Streaming at scale just got a lot more powerful and cost-effective. Our Q3 Confluent Cloud launch is packed with innovations to help you do more with less: reduce cloud networking costs while maintaining your security posture, scale effortlessly with boosted connection limits, and build production-ready agentic artificial intelligence (AI) applications with seamless tool integrations. 

Whether you're just moving data from a to b, optimizing mission-critical workloads, or building the next generation of intelligent apps, Confluent Cloud gives you the flexibility and tools to cost-effectively make it happen. Keep reading for a full breakdown to get the most out of our new cloud features.

• Maximizing Cost Savings on Serverless Clusters

• Building Event-Driven Agents Natively on Apache Kafka® and Apache Flink® With Streaming Agents (Open Preview)

• Custom Single Message Transforms and Custom Connectors

• Expanding Tableflow Functionality for Confluent Cloud and WarpStream Users

• Cluster Linking on Google Cloud

• Additional New Features and Updates

Maximizing Cost Savings on Serverless Clusters

Private Network Interface

We’re excited to announce that Private Network Interface (PNI) is now generally available for both Enterprise and Freight clusters on Amazon Web Services (AWS). PNI is a secure, low-cost private networking option built directly on AWS Elastic Network Interfaces (ENI), the same trusted AWS technology that powers popular services such as Amazon Elastic Kubernetes Service (EKS) and AWS Lambda.

We want to make Apache Kafka® cost-effective for any use case, and a huge piece of that cost optimization puzzle is getting networking costs under control. As throughput increases, your networking spend ramps up as well. While secure networking options like AWS PrivateLink provide strong isolation, their endpoint and per-GB charges can become a major expense for data-intensive applications. On the other hand, virtual private cloud (VPC) peering has traditionally been a cheaper way to move traffic but is often seen as less secure and introduces complexities like routing constraints.

That’s why we built PNI: to eliminate the difficult trade-off between cost and security that teams typically face with cloud networking when scaling their workloads.

PNI gives you access to Confluent Cloud through an ENI that lives directly inside your own VPC. By attaching an ENI from your account to Confluent Cloud services such as Enterprise and Freight clusters, you can apply your own security groups to manage all traffic—giving you full control with the tools and policies your team already knows. And because PNI avoids hourly and per-GB PrivateLink endpoint charges, it can significantly lower your cloud networking costs, including your AWS networking bill.

With PNI, you can:

• Lower networking costs by 20%-50% at scale with a secure and performant networking type built on top of fundamental AWS networking primitives

• Assert granular control over traffic to Confluent Cloud by applying your existing security groups and familiar network policies directly to a dedicated ENI within your own VPC, maintaining the robust security you’d expect from PrivateLink

• Seamlessly transition to cost-effective PNI without disrupting existing PrivateLink setups with dual connectivity support for both networking types on Enterprise clusters

As part of our release of PNI, we’re also lowering our throughput prices per GB of data read or written for clusters using this networking type: 

• Enterprise clusters on PNI: 20% throughput cost reduction ($0.05 → $0.04/GB)

• Freight clusters: 40% throughput cost reduction ($0.05 → $0.03/GB)

Check out our PNI blog to learn more about the technology and see how early adopters like Indeed are already using PNI to realize significant cost savings and strengthen the security of their data infrastructure.

Increased Client Connection Limits

We’re announcing a major boost in client connection capacity for Enterprise (new limits coming soon) and Freight clusters (new limits now available):

• Total connections per Elastic Confluent Unit for Kafka (eCKU): Now 18,000 (4x improvement for Enterprise, 2x improvement for Freight)

• Connection attempts per second per eCKU: Now 500 (2x improvement for both Enterprise and Freight)

These limits—on par with Dedicated clusters—are automatically applied to all new and existing clusters, with no manual changes needed. These higher connection limits are especially beneficial for microservices and hub-and-spoke architectures, where many clients connect concurrently. With more client capacity, you can avoid provisioning extra eCKUs just to handle connections, helping you optimize for both scalability and cost efficiency.

Building Event-Driven Agents Natively on Apache Kafka® and Apache Flink® With Streaming Agents (Open Preview)

As enterprises adopt agentic AI to automate more complex decisions, the need for agents that can act on real-time data is growing rapidly. But building production-ready agents is challenging:

• It often means stitching together disjointed systems for data processing, model inference, and orchestration, introducing fragility, latency, and security risks.

• Without access to fresh, contextualized information, agents can’t make effective decisions.

• Engineering teams may use separate tools and languages to work with AI models and data processing pipelines, leading to fragmented workflows.

These challenges result in inefficiencies, slower delivery, and difficulty adapting as new AI tools and techniques emerge. Streaming Agents on Confluent Cloud, now in Open Preview, solves these challenges by bringing agentic AI directly into your stream processing pipelines.

Built on fully managed Apache Flink® and Apache Kafka®, Streaming Agents enables you to build, deploy, and orchestrate event-driven agents that act on real-time data. Unlike agents created using other frameworks, Streaming Agents are deeply embedded in the infrastructure, with access to fresh contextualized data from real-time sources and the ability to adapt as conditions evolve and to communicate with other agents and systems. By bringing agentic AI directly into stream processing pipelines, Streaming Agents helps teams reduce complexity, iterate faster, and deliver intelligent, context-aware automation across an organization.

Streaming Agents enables you to build production-ready agentic AI applications by:

• Integrating seamlessly, with built-in support for connecting to models, tool calling with Model Context Protocol (MCP), contextual search, and data enrichment from external systems

• Accessing fresh context from unifying data processing and agentic AI workflows, enabling agents to operate dynamically on real-time events

• Ensuring that agents are trustworthy and secure, maintaining full visibility and control, with secure connections, role-based access control (RBAC), and governed, traceable event flows

Streaming Agents is comprised of several new features in Open Preview, including:

• Tool calling with MCP: Invoke external tools (e.g., software-as-a-service (SaaS) apps, APIs, databases) in real time using Anthropic’s MCP, allowing agents to take action with full business context and traceable interactions.

• Connections: Define secure, reusable connections to models, vector databases, and MCP servers directly in Flink SQL, keeping sensitive credentials safe and centralizing connection management for large-scale deployments.

• External tables and search: Enrich streaming data with external context from REST APIs, JDBC-accessible databases like MySQL, Oracle, and PostgreSQL, and vector search databases like MongoDB Atlas, Elastic, and Pinecone—powering accurate decision-making and retrieval-augmented generation (RAG) with a complete, real-time view of your data.

Additional Streaming Agents features that are now generally available include:

• AI model inference: Work directly with AI/machine learning (ML) models as first-class resources in Flink by calling remote models (e.g., OpenAI, SageMaker, Vertex AI) using familiar SQL syntax, reducing the need for specialized ML tools and languages.

• Embeddings: Work directly with embedding models in Flink SQL and continuously turn unstructured enterprise data into vector embeddings. As a result, you can enable RAG by creating a real-time, contextualized, and trustworthy knowledge base that can be coupled with prompts at inference time to generate better large language model (LLM) responses.

• Built-in ML functions: Use time-series forecasting and anomaly detection SQL functions, built on top of popular ML models like ARIMA, to derive real-time insights and make smarter decisions faster.

To learn more about Streaming Agents, check out the quickstart, demo video, and the deep dive blog.

Custom Single Message Transforms and Custom Connectors

Custom Single Message Transforms (SMTs)

We know many of you want to use fully managed connectors but need support to bring over your custom SMTs without any maintenance or operational hurdles. That’s why we’re introducing Custom SMTs for Confluent Cloud, which enables you to bring your own message transformation logic to fully managed connectors. SMTs are lightweight Java plugins that enable real-time message modification within Kafka Connect, allowing customers to perform essential data transformations such as format conversions without adding extra infrastructure. With support for both pre-packaged and in-house custom-built SMTs, Custom SMTs make it easier to adapt fully managed connectors to fit specific business and compliance needs.

With Custom SMTs, you can:

• Upload, execute, and manage your own custom SMT code, ready for utilization with any new or existing fully managed connector within the cloud environment

• Meet data security and regulatory requirements by enabling transformation or processing before data is stored in Kafka topics

• Seamlessly migrate from self-managed to fully managed connectors with comprehensive support for your custom SMTs, alleviating the burden of ongoing maintenance and operation

Custom SMTs for Confluent Cloud are currently available on AWS across all major private networking types, including AWS PrivateLink, VPC peering, and Transit Gateway. With Custom SMTs, developers get the customization they need, freeing them to focus on building—not maintaining—their streaming data infrastructures.

Custom Connectors on Google Cloud

We’re excited to announce that Confluent Cloud now supports custom connectors on Google Cloud, available in 13 regions. With this addition, you can deploy and run custom connectors on Confluent across all three major cloud providers. Developers can now deploy Kafka Connect-compatible custom connectors directly in Confluent’s fully managed environment—eliminating the need to operate separate infrastructure. This makes it easy to integrate proprietary, legacy, or specialized systems while allowing teams to focus on delivering value from their data.

Expanding Tableflow Functionality for Confluent Cloud and WarpStream Users

Tableflow Upsert Support (Coming Soon)

Tableflow is evolving beyond append-only functionality with the introduction of upsert support. Upsert functionality is a database operation that updates an existing row in a Tableflow table if a specified value already exists in a table and inserts a new row if the value does not exist. With upsert support, users will be able to create highly efficient, performant data tables that eliminate redundancy and streamline file reconciliation when powering real-time analytics use cases.

Here’s how it will work: Tableflow upsert mode will allow you to use user-defined keys in Kafka to add primary key constraints to your Tableflow table. By doing so, you’ll enable the system to track changes for each key. Instead of overwriting data, Tableflow will create a delete file that captures the changes made. Over time—or once enough changes have accumulated—Tableflow will automatically compact the files, merging updates and rewriting them to maintain performance and data integrity. And if you're using a compute engine that supports “merge on read” queries, it can apply these changes instantly at query time, so your results will always reflect the latest state of your data.

With Tableflow upsert mode, you can:

• Simplify row-level table modifications by using user-defined keys as primary keys to automatically track and apply updates and deletions

• Keep tables performant and cost-effective during updates through “merge on read” operations, balancing query timeliness with the efficiency of merging multiple changes during compaction

• Facilitate compatibility with any compute engine supporting "merge on read" query mode for real-time change reconciliation during data querying

WarpStream Tableflow (Early Access)

WarpStream Tableflow is now in Early Access, bringing the Tableflow experience to the WarpStream ecosystem and automating the creation of fully managed Apache Iceberg™ tables from any Kafka-compatible source, including open source Kafka, Confluent Platform, and WarpStream itself. There’s one key difference: Your data is stored in your own cloud object storage, with a zero-access, BYOC-native architecture that ensures that WarpStream never accesses your raw data.

Similar to Confluent Cloud, WarpStream Tableflow simplifies the process of representing streams as tables to power analytics and lakehouses without the need for complex ingestion pipelines or extra infrastructure. It handles complex table management operations like compaction, schema evolution, and metadata management, so your data remains fresh, queryable, and easy to manage.

WarpStream Tableflow operates across the three major cloud providers and on-premises environments. Reach out to your account team to take part in the Early Access program.

Cluster Linking on Google Cloud

We’re excited to expand the reach of cluster linking, with support for private networking on Enterprise and Dedicated clusters—including Private Service Connect (PSC) and VPC peering—on Google Cloud. With this release, you can now link clusters across Google Cloud, AWS, and Microsoft Azure over private networking, enabling cross-cloud Cluster Linking without the need for VPN tunnels, IP filtering, or custom setups.

Now, cluster linking is available across all three major cloud providers, for both public and private networking configurations and for both Enterprise and Dedicated clusters. Whether you’re enabling secure data sharing or building for disaster recovery, cluster linking simplifies Kafka replication while ensuring that you meet your security requirements.

Additional New Features and Updates

WarpStream Multi-Region Clusters

WarpStream Multi-Region Clusters, available on AWS, lets you deploy a single cluster across multiple regions, delivering the resilience of a multi-region architecture without the cost or complexity of managing separate deployments. WarpStream handles the heavy lifting behind the scenes, eliminating the operational overhead of managing separate clusters or handling failover logic yourself.

With fully automated, transparent failover and Recovery Point Objective (RPO) guarantees, you can achieve zero data loss (i.e., RPO = 0) even in the event of a full regional outage. Backed by a 99.999% uptime SLA, this architecture is built to meet the most demanding disaster recovery and compliance requirements so that you can focus on building always-available real-time applications.

Compaction Support for Freight Clusters

Freight clusters now support compaction, unlocking a wider range of real-time use cases that require retaining only the latest value for each message key in a topic, such as change data capture (CDC) and restoring state after system failure. Compaction support makes it easier to build efficient, stateful applications on Freight clusters, with the added benefit of improved compatibility for connectors that work best with compacted data. Support for transactions is also on the way in the coming weeks, further expanding what’s possible on Freight.

New Fully Managed Connectors

Couchbase Source and Sink Connectors

The fully managed Couchbase Source and Sink connectors offer an optimized and effortless way for organizations to connect their Couchbase NoSQL databases with Confluent Cloud. Both connectors offer at least-once delivery and support Private Link on AWS and Azure. The Couchbase Sink Connector also supports Client-side field level encryption (CSFLE) for sensitive data. To learn more, read our partner blog.

ClickHouse Sink Connector

The fully managed ClickHouse sink connector moves data from a Kafka topic to the ClickHouse database. This connector offers exactly-once delivery, supports AWS PrivateLink and Google Cloud PSC, and also supports CSFLE for sensitive data. To learn more, read the blog.

MariaDB CDC Source (Debezium) Connector

The fully managed Maria DB CDC source (Debezium) connector can obtain a snapshot of the existing data in a MariaDB database, then monitor and record all subsequent row-level changes.

Enhanced Private Networking for Flink and Schema Registry

Confluent Cloud Network (CCN) routing support for Flink and Schema Registry on Azure

Earlier this year, we announced the availability of CCN routing support for Flink and Schema Registry on AWS. Now, we're excited to extend this capability to Azure. With CCN routing, you can securely connect to Flink and Schema Registry using your existing Private Link-based CCN infrastructure, without the need for separate PrivateLink Attachments (PLATTs).

This dramatically reduces setup complexity, especially if you’re already using Dedicated clusters with private networking. Support for Azure Virtual Network Peering-based CCNs will be released in the following weeks, further expanding private networking options on Azure.

CCN routing for Flink is available on Azure in all regions where Flink is supported. CCN routing for Schema Registry is available on Azure in select regions, with support for additional regions coming in the following weeks.

Private Networking on Confluent Cloud for Flink and Schema Registry on Google Cloud

We're pleased to extend private networking support for Flink to Google Cloud for both Enterprise and Dedicated clusters, making it possible for users to query data in Kafka clusters in PSC and VPC peering networks. With this release, private networking for Flink is available on all three major cloud providers, enabling users to strengthen security and compliance across their Flink deployments. See the list of currently supported cloud regions in this documentation.

We’re also pleased to extend private networking support for Schema Registry to Google Cloud through the utilization of PLATTs for private connectivity. Now, customers can connect to Schema Registry privately within their Google Cloud VPC, eliminating public internet exposure for sensitive data. See the list of currently supported cloud regions in the documentation.

Improved Error Handling and Observability for Flink

Custom error handling for Flink gives you fine-grained control over how to handle deserialization errors in your pipelines. Choose from three error-handling modes (fail, ignore, or log) on a per-table basis to prevent bad records from disrupting your data flow. When using log mode, problematic records and error details are automatically sent to a dead letter queue (DLQ) table for later inspection. Custom error handling improves the resilience and manageability of Flink statements, ensuring that your applications continue running smoothly even when encountering unexpected or malformed data.

Query Profiler for Flink helps you analyze and improve the performance of your Flink statements with detailed, real-time metrics to help you identify and resolve performance impediments. It displays a real-time job graph of your running statement, complete with task-level metrics like throughput, state size, and more. Color-coded indicators help you quickly identify performance bottlenecks while detailed panels provide deeper insights per task. This self-service tool makes it easier to debug issues and optimize performance.

Start Building With New Confluent Cloud Features

If you’re new to Confluent, sign up for a free trial of Confluent Cloud and create your first cluster to explore the new features. New sign-ups receive $400 to spend within Confluent Cloud during their first 30 days. Use the code CCBLOG60 for an additional $60 worth of free usage.*

The preceding outlines our general product direction and is not a commitment to deliver any material, code, or functionality. The development, release, timing, and pricing of any features or functionality described may change. Customers should make their purchase decisions based on services, features, and functions that are currently available.

Confluent and associated marks are trademarks or registered trademarks of Confluent, Inc.

Apache®, Apache Kafka®, Apache Flink®, Flink®, Apache Iceberg™️, and the respective logos are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. No endorsement by the Apache Software Foundation is implied by using these marks. All other trademarks are the property of their respective owners.