Level Up Your Kafka Skills in Just 5 Days | Join Season of Streaming On-Demand
The journey from data mess to data mesh is not an easy one—that’s why we’ve written a new ebook as a practical guide to help you navigate the challenges and learn how to successfully implement a data mesh using Confluent Data Streaming Platform, including:
How to best put the four principles of data mesh into practice, leveraging Confluent connectors, stream processing, and Stream Governance.
How to turn raw real-time data into high-quality data products and make them discoverable and accessible across your organization.
How to develop a winning strategy to gain broad adoption of your streaming data mesh.
To put the data mesh adoption journey into perspective, it’s worth taking a step back to see how we got here. Data has traditionally resided in two domains: the operational estate, which includes databases, CRM, ERP apps, and billing systems for daily business operations; and the analytical estate, comprising data warehouses for after-the-fact analysis to inform business decisions, create marketing campaigns, financial analytics, dashboards, and more.
Over the years, there has been a proliferation of new business applications, SaaS offerings, and specialized tools, alongside a hybrid operating model integrating cloud services with on-premises systems. At the same time, development teams are transitioning from building monolithic applications to event-driven microservices, necessitating seamless integration for real-time communication.
This complexity extends to the analytical domain. As organizations seek to consolidate data across silos of operational sources for various stakeholders, what was a handful of batch systems, messaging middleware, and APIs has evolved into a data mess—a growing sprawl of point-to-point integrations that hinder the ability to build innovative new applications.
To overcome these data barriers, organizations need a more effective approach to harnessing data—making shared data discoverable, trustworthy, and secure so that other teams can make good use of it. This strategy aligns with the principle of data as a product, a key tenet of the data mesh framework.
Data mesh is a transformative approach to IT infrastructure, incorporating people, processes, and technology. It decentralizes data architecture, empowers individual teams, and pushes for greater agility and collaboration in managing and deriving value from diverse, domain-specific data products.
Zhamak Deghani’s Data Mesh Principles and Logical Architecture explains these four principles in detail:
Domain-driven ownership of data
Data as a first-class product
Federated computational governance
Self-service data platform
If you have decided to adopt a data mesh—what’s next? What are the key milestones? How do you expand upon your tooling, frameworks, and infrastructure to enable self-service data for the wider organization, accelerating your velocity in tackling data challenges?
In the ebook, we document the data mesh adoption journey as a five-stage maturity model seen above. The details around organizational processes for internalizing data mesh principles and adopting best practices can be found here, but the key to successful data mesh is focusing on data product value and developing repeatability and efficiency. To avoid the pitfalls of data meh and data mess, we need to demonstrate the end-to-end value of data products and ensure that data products are always designed in a way so that they may be easily evolved.
Within a data mesh, data products are sourced across various domains within your organization, utilized by different stakeholders for bespoke purposes. This is a departure from centralized data organization models. A successful data mesh should make it easy to access, use, and publish data products, with strong governance and lineage tracking across domains.
How do you apply product thinking to your data?
Consider all the entities in your business—customers, purchases, inventory, shipments… each of these is a data product. Instead of querying data from rows in a database, data products are live. They are continuously enriched, governed, and shared so that every team can have frictionless self-service access to trustworthy data assets and derive value from data the moment it’s created.
Data products can be combined for greater reuse, such as joining customer profiles, web logins, and transactions to create a holistic 360-degree view of each customer interaction. This can also enable the dynamic generation of threat scores to prevent fraudulent activities. Or, you can blend live data products to enhance your customer loyalty program and delivery management systems. In this way, data products are reusable for unlimited operational and analytical use cases.
Unlike existing approaches where each team and use case requires investing significant time in identifying and searching for data, formatting it appropriately, or building custom datasets and data pipelines, Confluent enables your teams to seamlessly leverage high-quality data products:
Pre-built, fully managed source and sink connectors help you build streaming data pipelines that unlock real-time data flows, breaking down data silos across hybrid and multicloud environments.
Stream processing enables you to transform, combine, enrich, and clean data in flight to create ready-to-use data products.
Stream Governance ensures your data’s security, quality, and compliance.
Data Portal allows you to share, discover, and access data products across your organization.
Confluent as a foundation for data mesh allows your teams to independently and collaboratively derive greater value, promoting autonomy, agility, and adaptability for evolving business needs. As a result, you not only reduce operational costs but also mitigate the risks and challenges associated with data governance.
The success of a data mesh heavily depends on treating data as a product, applying rigorous standards, and fostering a consistent governance structure. Confluent Data Streaming Platform is well suited for data mesh given its ability to bridge operational and analytical domains, making real-time, governed, and enriched data available for use in the right place, in the right format. The self-service aspects of the platform, particularly in a cloud-native context, plays a pivotal role in enabling quick ideation to application deployment. In the ebook, we detail this implementation, with emphasis on scalability, flexibility, and ease of access for consumers. Confluent provides capabilities essential for building a comprehensive data mesh:
Data mesh heralds a shift from centralized data approaches to a federated, self-serve paradigm, and you can leverage Confluent Data Streaming Platform as the foundation for effective data product delivery.
Over the years with customer implementations of data mesh, we’ve distilled our experiences into a practical how-to guide for architects and technology executives who are looking to implement a data mesh.
The ebook covers how to use specific Confluent products to put each of the four principles into practice at your organization as well as best practices for making the organizational and technical changes needed to successfully gain broad adoption of your data mesh.
A data mesh is more than a technological solution; it is a new framework for sharing data within and even beyond an organization. Success involves implementing best practices focused on interoperability without relying on centralized control, continually adapting to your organization's unique needs.
We encourage you to read The Builder’s Guide to Streaming Data Mesh and connect with us to discuss how data mesh can look at your organization. We also have a webinar with Q&A to walk through the implementation steps. Don’t miss it, join the webinar today:
This blog explores how cloud service providers (CSPs) and managed service providers (MSPs) increasingly recognize the advantages of leveraging Confluent to deliver fully managed Kafka services to their clients. Confluent enables these service providers to deliver higher value offerings to wider...
With Confluent sitting at the core of their data infrastructure, Atomic Tessellator provides a powerful platform for molecular research backed by computational methods, focusing on catalyst discovery. Read on to learn how data streaming plays a central role in their technology.