Level Up Your Kafka Skills in Just 5 Days | Join Season of Streaming On-Demand
In today's rapidly evolving technological landscape, efficient communication between platforms and merchants is crucial for e-commerce success. For Extend, a modern product protection platform, ensuring seamless backend communication with their merchant partners was a top priority. Each merchant across a number of industries (e.g., auto, electronics, jewelry, and furniture, etc.) had a constantly changing product catalog, requiring real-time updates. To address this challenge and improve their engineering experience, Extend aimed to decouple their backend infrastructure and transition toward an event-driven architecture.
Extend initially adopted a serverless-first approach using Amazon Web Services (AWS) technologies to prioritize their minimum viable product (MVP) and minimize operational overhead. This approach enabled them to focus on solving customer problems without the burden of managing physical hardware. However, as their platform grew and evolved, they realized the need for a more robust and scalable solution to stream events in real time.
Powered by Kora Engine, the Apache Kafka® engine built for the cloud, Confluent Cloud emerged as the perfect fit for Extend's serverless-first platform vision. By leveraging Kafka for their event-driven architecture, Extend sought to tackle its technical challenges while reducing overhead, improving time to market, gaining scalability, and leveraging automation tools.
One of the primary reasons Extend chose Confluent Cloud was to eliminate the burden of overseeing the technical infrastructure required for self-managing Kafka. They wanted their engineers to focus on utilizing Kafka's capabilities to solve technical challenges, rather than spending time managing clusters, nodes, brokers, and networks. Confluent's cloud-native, fully managed solution freed Extend from operational overhead, allowing their engineers to focus on delivering immediate business value to customers and internal users.
Before adopting Confluent Cloud, Extend relied on Amazon Simple Notification Service (Amazon SNS) and Amazon Simple Queue Service (Amazon SQS) which were used by multiple services on their platform to support all asynchronous communication between services. However, the data volume generated significant throughput spikes, especially during peak periods such as Black Friday or Cyber Monday. With adoption of Confluent Cloud, Extend's engineers do not need to log in to the Confluent Cloud UI. Confluent manages all the clusters, nodes, brokers, and networks in the background, providing a decoupled approach commonly used by larger companies.
One of the key advantages of utilizing Kafka was its ability to provide ordered events, replayable events, and retention of events. These features addressed the challenges Extend faced with ordered events and duplication when using SNS/SQS. As Extend moved toward an event-driven architecture, they could leverage Kafka's capabilities to ensure consistent and reliable communication between their services and merchant partners.
Merchants partnering with Extend often use tools like NetSuite, Shopify, and BigCommerce to manage their product catalogs. Extend needed to handle massive amounts of data, and their previous stack struggled with handling the throughput. Extend achieved the desired scalability and real-time data processing capabilities by adopting a serverless architecture and leveraging the combined power of AWS and Confluent.
Extend's adoption of Confluent's fully managed data streaming platform allowed them to fulfill the core principles of their technology vision and lay the foundation for future strategic initiatives:
Serverless first: By using AWS serverless technologies and Confluent Cloud, Extend could prioritize solving customer problems without the need to manage physical infrastructure.
Event-driven architecture: Transitioning toward an event-driven architecture enabled Extend to decouple their services and facilitate asynchronous communication, improving the engineering experience.
Least privileged access: Confluent's tools and integrations enabled Extend to adhere to the principle of least privileged access, ensuring secure and controlled access to their backend systems.
Automation and efficiency: The partnership with AWS and Confluent allowed Extend to automate its communication layer and reduce operational overhead, enhancing efficiency and time to market.
Data mesh: The adoption of event-driven architecture laid the groundwork for implementing data mesh principles, enabling Extend to leverage data streaming technologies and integrate data seamlessly with merchant partners.
Try Confluent for free on AWS Marketplace
Extend combined custom CloudFormation templates and CDK to simplify the provisioning and management of their Kafka resources with Confluent Cloud APIs. This automation allowed them to auto-provision environments, clusters, topics, API keys, and access controls. Additionally, they utilized AWS Lambda triggers to consume data from Kafka topics without requiring long-lived infrastructure in their environments.
Instead of creating a reference architecture, Extend developed internal libraries and packages to simplify creating new topics and clusters. These tools automated the setup of permissions, access controls, and clusters, providing engineers with a streamlined experience for producing events without being burdened by technical details. While a reference architecture is still a work in progress, it will soon provide engineers who prefer copy-paste code with a working, proven example.
By combining the capabilities of AWS Lambda’s serverless compute with Confluent’s serverless Kafka offering, Extend successfully codified and automated its infrastructure, making Kafka an essential component of their event-driven architecture. Engineers could rapidly spin up new topics and leverage Kafka for asynchronous communication, boosting productivity and enabling efficient backend communication with merchant partners.
From a business perspective, Extend's adoption of Confluent Cloud resulted in significant benefits. They eliminated the need to hire full-time employees to maintain technical infrastructure, reducing their total cost of ownership. Thanks to the time saved from managing Kafka infrastructure, the teams could focus more on delivering value to customers. Additionally, by shifting from capital expenditures (CapEx) to operating expenses (OpEx), Extend optimized its operational costs and improved overall efficiency.
“We don't have to hire additional engineers to maintain infrastructure. They don’t have to spend time managing Kafka brokers, clusters, nodes, etc. We don't have to have anybody who manages that full time. It's handled for us as a managed service from Confluent.” — Jonathan Kropp, Director of Architecture, Extend
By streamlining backend communication with merchants through the power of Confluent Cloud and AWS, Extend successfully realized its technical vision of a serverless-first, event-driven architecture. With the ability to seamlessly integrate and process real-time data, Extend continues to redefine the product protection industry and provide exceptional experiences for its customers and merchant partners.
Data silos often hinder an organization's ability to make quick and informed decisions, which impacts business and customer experience. Extend's use cases illustrate that Confluent's integration with major cloud providers such as AWS empowers companies to leverage their critical data and gain an edge over their competition.
Check out the AWS “Voice of the Customer” video with Extend where Jonathan Kropp, Director of Architecture at Extend, discusses how they codified and automated their infrastructure by pairing Confluent's cloud-native, serverless offering with AWS Lambda to provide a modern product protection platform for consumers.
Try Confluent for free on AWS Marketplace
This blog explores how cloud service providers (CSPs) and managed service providers (MSPs) increasingly recognize the advantages of leveraging Confluent to deliver fully managed Kafka services to their clients. Confluent enables these service providers to deliver higher value offerings to wider...
With Confluent sitting at the core of their data infrastructure, Atomic Tessellator provides a powerful platform for molecular research backed by computational methods, focusing on catalyst discovery. Read on to learn how data streaming plays a central role in their technology.