Setting Up Secure Networking in Confluent with Azure Private Link

We’re happy to announce that Confluent Cloud, our cloud-native service for Apache Kafka^®, now supports Azure Private Link for secure network connectivity, in addition to the existing Azure Virtual Network (VNet) peering and secure internet connectivity options. Azure Private Link is supported on Dedicated clusters whether you procure Confluent Cloud directly from Confluent or through the Azure Marketplace.

Azure Private Link is an Azure proprietary networking service that allows one-way secure connection access from your VNet to both Azure and third-party services. Now you can create a new Dedicated cluster in Confluent with Private Link enabled, set up a private endpoint in your Azure VNet, and securely connect to Confluent’s platform for data in motion from your VNet.

Multiple organizations are already using Azure Private Link for their Confluent deployments, and we’re excited to now offer this capability to all Confluent customers using Dedicated clusters. Continue reading to learn more about this networking option or skip to instructions for setup in your account.

Why choose Private Link?

Enterprises use Private Link for its unique combination of security and simplicity.

For many companies, a multi-layer data security policy starts with addressing network attack vectors exposed to the public internet. Security breaches, DDOS attacks, spam, and other concerns can be prevented by blocking internet access to key resources like Kafka clusters. VNet peering—where two parties share network addresses across two networks—has historically been a common solution for private network connectivity, but it has its downsides.

VNet peering requires both parties to coordinate on a unique IP address block for communication between the networks. Many companies, especially large enterprises, have limited IP space, so finding an available IP address block can be challenging and requires a lot of back and forth between teams and between the peering parties. This can be especially painful in large organizations with hundreds of networks connected in a sophisticated topology. Applications that need access to Kafka are likely spread across many networks, so peering them all to Confluent is a lot of work.

Once a VNet peering connection is set up, each party has access to the other network—that’s what connectivity means—but this isn’t always desirable. Confluent users want their clients to initiate connections to Confluent Cloud but restrict Confluent from having access back into their network.

Private Link enables network-level security without the downsides of VNet peering. Confluent exposes Private Link service alias(es) for each new cluster, for which customers can create corresponding private endpoints in their own Azure VNets. Users don’t have to juggle with IP address blocks for Confluent because their clients connect using the private endpoint with an associated IP address whose scope is local to the user’s VNet. It’s a one-way connection from the user to Confluent, so there’s less surface area for the network security team to keep secure. Making dozens or hundreds of Private Link connections to a single Confluent cluster doesn’t require any extra coordination with Confluent nor within your organization.

With all these benefits, it’s not surprising that Azure recommends Private Link as the best method for private connectivity between Azure VNets.

Behind the scenes

Supporting Private Link in Confluent has been a major effort. Last year, we rolled out support for AWS PrivateLink by introducing a new networking stack to help expose Confluent through AWS PrivateLink using AWS Network Load Balancers. We also introduced self-serve customer AWS account registration with Confluent to enable complete automation of PrivateLink interface endpoint creation from the customer VPC.

We have now extended Private Link support to Azure by adapting the networking stack to run on Azure, and we have integrated it with Azure Standard Load Balancers and Azure Private Link Service. The Confluent and Azure Private Link solution also includes self-serve Azure subscription registration with Confluent to enable complete automation of private endpoint creation from the customer VNet. The outcome: Spin up a Dedicated cluster in Confluent and get an Azure Private Link Service alias in minutes directly through the Confluent UI, totally self-serve, so your clusters can be up and running in no time.

Get started with Azure Private Link in Confluent Cloud

Connect to Confluent securely from your Azure subscription using Azure Private Link Service and enjoy the unique benefits of this connectivity.

1. Within the Confluent UI, create a new Dedicated cluster in Azure. Choose the Azure region that you want to run in, whether the cluster should run in one or multiple availability zones, the cluster’s capacity, and Private Link as the networking option.
   
   
2. After the cluster is provisioned, you’ll receive an email and see an alert in the Confluent UI to finish setting up the Private Link connection. For security, Confluent requires you to register the subscription IDs that you’ll connect from so we can guarantee that only your organization can make Private Link connections to your Confluent cluster. You can connect from any or all of the VNets in the Azure subscriptions you register, so even if you have Kafka clients spread across dozens of VNets, you can securely connect them all to Confluent.
3. Set up private endpoints in your Azure subscription. Azure Private Link actually consists of two parts: the Private Link Service exposed by Confluent and private endpoints that you configure within your Azure subscription. Confluent offers a Terraform script to automate the setup of your private endpoints, as there are quite a few steps. If Terraform is not for you, there is also detailed documentation that you can follow. You will be prompted to provide the Private Link Service alias when creating your private endpoint, as shown below.
4. Set up a private DNS zone in your Azure subscription and add DNS records to route the Confluent Cloud traffic to private endpoints. Follow the detailed steps in the documentation to provision DNS.
5. You’re ready to go!

As you can see, with just four simple steps, Confluent and self-serve Azure Private Link support provides an enhanced user experience with secure and hassle-free connectivity. We would love to hear about your experience and encourage you to join our Community Forum and connect with industry experts!

Join some of the most security-conscious, highly-regulated companies in the world using Azure Private Link on Confluent Cloud, and get started for free. When you sign up, you’ll receive $400 to spend within Confluent Cloud during your first 60 days. You can also use the promo code CL60BLOG to get an additional $60 of free Confluent Cloud usage.*

Start Free