Kafka in the Cloud: Why it’s 10x better with Confluent | Find out more
Are you looking for a way to run AWS services on premises in your own datacenter? I am excited to share today that we have completed validation of support for running Confluent Platform on AWS Outposts. AWS Outposts delivers fully managed, configurable compute and storage racks built with Amazon Web Services (AWS) designed hardware that allows you to run instances as you would on Amazon EC2 locally in your datacenter or co-location facility. You get all these resources locally while seamlessly connecting to a broad array of cloud-based AWS services.
We are also excited to announce that we are AWS Outposts Ready partner certified—one the highest designations that an Amazon Partner Network (APN) Partner can achieve. This means that Confluent Platform has been certified for integration with AWS Outposts, giving you the assurance that can run your production workloads on Confluent Platform locally and securely.
We completed testing of Confluent Platform on Outposts and can now connect various on-premises event sources like legacy databases, proprietary storages, and/or monolithic applications to AWS, serving as a data hub for hybrid designs. Confluent Platform is an enterprise-ready platform that complements Apache Kafka® with advanced capabilities designed to help accelerate application development and connectivity, enable digital transformations through stream processing, simplify enterprise operations at scale and meet stringent architectural requirements. The advantages of running Confluent Platform on AWS Outposts include:
In collaboration with AWS, we are able to provide a true hybrid experience with a unified control plane for AWS infrastructure, as well as a unified event streaming platform to build out your next-generation event-driven applications. Thus, you can overcome challenges that exist due to data locality requirements and running complex hybrid scenarios between on-premise and AWS services.
Next, let’s walk through how we set up our environment.
This section covers how we deployed Confluent Platform to Outposts during our validation process. I will cover the connectivity you will need to have between your Outposts instances, AWS, and the internet. Then, we will delve into building a custom Amazon Machine Image (AMI) on top of Ubuntu Linux that will be used as a basis for all of the instances we deploy. Next, we will run through a few updates that you need to make to the CloudFormation template supplied below. Finally, we will close with how to verify that your deployment was successful.
Let’s first discuss the connectivity that EC2 instances running in your Outposts will need:
Follow these steps to create the customer Ubuntu 18.04 AMI:
sudo apt-get update -y
sudo apt install python3-pip -y
pip3 install ansible --user
vi .bashrc
export PATH=$PATH:$HOME/.local/bin
source .bashrc sudo cp .local/bin/ansible* /usr/bin
sudo su pip3 install ansible --user
pip3 install awscli --user sudo cp .local/bin/aws* /usr/bin
sudo su pip3 install awscli --user
pip3 install boto3 --user
sudo su pip3 install boto3 --user
sudo apt-get install python-pip -y sudo pip install https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-latest.tar.gz sudo cp /usr/local/init/ubuntu/cfn-hup /etc/init.d/cfn-hup sudo chmod +x /etc/init.d/cfn-hup sudo update-rc.d cfn-hup defaults sudo service cfn-hup start
logout
vi /home/ubuntu/.bashrc
export ANSIBLE_SSH_USER=ubuntu
source .bashrc
sudo su
vi /root/.bashrc
export ANSIBLE_SSH_USER=ubuntu
source /root/.bashrc
rm .ssh/authorized_keys sudo shred -u /etc/ssh/*_key /etc/ssh/*_key.pub
sudo su rm .ssh/authorized_keys shred -u /etc/ssh/*_key /etc/ssh/*_key.pub
sudo passwd -l root
rm .bash_history
sudo su rm .bash_history history -c && history -w && exit
history -c && history -w
"Mappings": { "RegionMap": { "ap-northeast-1" : { "AMIId" : ""}, "ap-south-1" : { "AMIId" : ""}, "ap-southeast-1" : { "AMIId" : ""}, "ap-southeast-2" : { "AMIId" : ""}, "ca-central-1" : { "AMIId" : ""}, "eu-central-1" : { "AMIId" : ""}, "eu-west-1" : { "AMIId" : ""}, "eu-west-2" : { "AMIId" : ""}, "sa-east-1" : { "AMIId" : ""}, "us-east-1" : { "AMIId" : ""}, "us-east-2" : { "AMIId" : ""}, "us-west-2" : { "AMIId" : "ami-0081b1729374885e9"} }
"InventoryInstanceType" : { "Description" : "Choose EC2 instance type for Ansible Inventory Node", "Type" : "String", "Default" : "t2.micro", "AllowedValues": [ "t2.micro", "t2.medium", "t2.small", "m5.large" ] }, "BrokerInstanceType" : { "Description" : "Choose EC2 instance type for Kafka Broker", "Type" : "String", "Default" : "m4.large",
$ vi /var/log/cloud-init-output.log
If there are any errors in the parameters entered, the cloud-init-output.log will have a log saying that it failed to send an cfn-signal.
kafka-topics –create –topic test-dr-topic –bootstrap-server <BROKER IP>:9092 –replication-factor 3 –partitions 1
kafka-avro-console-producer --broker-list <BROKER IP>:9092 --property schema.registry.url=http://localhost:8081 --property key.converter=StringConverter --topic test-dr-topic --property value.schema='{"type":"record","name":"myrecord","fields":[{"name":"f1","type":"string"}]}'
{"f1": "value1-a"} {"f1": "value2-a"} {"f1": "value3-a"}
Press Ctrl-C. Consume messages:
kafka-avro-console-consumer --from-beginning --topic test-dr-topic --bootstrap-server <BROKER IP>:9092 --property print.key=false --property schema.registry.url=http://localhost:8081
You should see three messages consumed:
{"f1":"value1-a"} {"f1":"value2-a"} {"f1":"value3-a"} Processed a total of 3 messages
Press Ctrl-C. Delete your test topic through the Confluent Control Center GUI.
And that’s a wrap!
AWS Outposts provides a new paradigm for running on-premises workloads together with Confluent Platform for truly event-driven, hybrid, and mission-critical use cases. We covered many of the advanced features of AWS Outposts and walked through the exact process used to get Confluent Platform up and running for our AWS Outposts Ready partner certification. We hope this helps you on your path to event streaming.
If you’d like more details, I encourage you to learn more about AWS Outposts and get started with Confluent Platform today.
We covered so much at Current 2024, from the 138 breakout sessions, lightning talks, and meetups on the expo floor to what happened on the main stage. If you heard any snippets or saw quotes from the Day 2 keynote, then you already know what I told the room: We are all data streaming engineers now.
We’re excited to announce Early Access for Confluent for VS Code. This Visual Studio integration streamlines workflows, accelerates development, and enhances real-time data processing, all in a unified environment. This post shows how to get started, and also lists opportunities to get involved.