Capturing tech trends has become a bit tricky these days: whatever industry you’re in, uncertainty abounds. Planning has become harder, but businesses are finding new ways to innovate and respond quickly to fast-changing market conditions. And at Current 2022 last year, data streaming professionals from around the industry gathered in person and virtually to share ideas and solve problems together.
Across organizations and industries, some common areas of challenges and growth rose to the surface. For starters, the technology that’s succeeding these days is expected to tackle reality in, well, real time. That’s because businesses are moving fast to give users what they demand, and the old ways of capturing and storing data aren’t cutting it anymore.
Keep reading to learn what we saw at the first broad data streaming industry event—and what to keep an eye on for 2023 and beyond.
And if you’re interested in understanding the basics of data streaming and how it can transform your business, check out our Data Streaming Resources Hub. Here you’ll find all of the latest explainer videos, case studies, and industry reports on data streaming! Happy learning.
We’ve moved from “digital transformation” as the buzzword du jour into a more nuanced understanding of what transformation looks like at an enterprise level. It’s about data now, with some key capabilities emerging: data should be shareable across an organization, not just for developers. More and more, it needs to be processed by streaming, not batch, to keep ahead of what users need. It also needs to be governed and allow for self-service access, so emerging data technology has to be easy to use.
“In the past, data was a fixed point in time to be stored. That got us here,” Confluent co-founder and CEO Jay Kreps said during his Current keynote. “But reality isn’t some static fixed thing. Today’s use cases demand streaming data.”
Data transformation is what will allow teams to focus on outputs and business value, not infrastructure or service management. The ultimate goal? Data as a product, so that teams can access the data they need, when they need it, securely.
If data is going to truly power your business, it needs to be treated like the valuable asset it is. More than a technology shift, this is a mindset shift. It requires you to treat your data as if it were a high quality, ready-to-use product that’s instantly accessible across the organization. Then it’s consistent everywhere, which means that everyone is using the same data and taking advantage of the latest and greatest data.
When data is a first-class citizen, it helps operational systems serve customers better, analytical systems meet the demands of your stakeholders, and SaaS applications are always up to date. It’s governed, which means you can track where the data is coming from, where it’s going, and who has access to it. Keeping track of initial data quality is essential to data taking a leading role, as is tracking its lineage. Eventually, data assets are discoverable through contracts, so whoever needs access to the data in whatever format can easily subscribe and use it on-demand.
The result of applying this kind of product thinking to your data? It accelerates use case delivery and innovation.
Governance of all this data was a hot topic for Current attendees. Making data self-serve and accessible is foundational to the bigger goal of data as a product. The big question in data governance is: who is consuming what? It’s data, but it’s also all of the copies of that data.
“Right now, you can either go fast or be safe,” said Chad Verbowski, Senior Vice President of engineering at Confluent, during his Current keynote. “Accomplishing both will be key for business success.”
At the moment, according to IDC’s 2022 Data Trust Survey, respondents know how important trusted data is. In fact, more than 75% said that high data trust levels have a positive impact on customer satisfaction. At the same time, they’re still trying to build the infrastructure to make sure data is both trusted and broadly accessible. But only 17% of respondents have a complete architecture built for managing and controlling data. And streaming data is one of the least trusted sources of data.
Moving toward a data-as-a-product mindset will require broad trust, with companies using trusted platforms for broad streaming governance.
As older legacy data systems attempt to meet today’s real-time needs, data architectures and pipelines are showing the strain. Many businesses are grappling with a web of point-to-point systems that are hard to scale and maintain, especially in the real-time processing era.
Data sources, formats, and destinations all continue to grow, and those accessing and developing with data want both real-time and historical data to be available. But pipelines built for the batch era aren’t usually reusable or built for developers.
Traditional data pipelines tend to fall victim to five typical challenges:
They’re not real-time
They are centralized
The tools that build them lack the right governance capabilities
Developers are managing the infrastructure that’s running the pipelines
The tools that build them are inflexible
The good news? Building better pipelines is possible and will open up time for higher-level work for teams across IT.
One final trend that came up at Current: business and IT teams today need to be spending their time driving innovation, and not managing open-source tools like Kafka. Focusing on innovation, not infrastructure, is how businesses can get value from data and pull ahead of their competitors.
Building data pipelines is time-consuming and onerous for developers, taking time away from the truly interesting work. “My dream is to have enterprise topics and everything cataloged in Kafka,” said Pritha Mehra, CIO for the United States Postal Service, during her Current keynote. “Developers can just have a fun day making apps and not worrying about data.”
One thing is clear: there’s a lot to learn and explore, with so much potential for new ideas and ways of working in real time. As Gian Morlino of Imply said on the Current stage: “Streaming vs. batch is as big a shift as mobile phones were. When it comes to streaming, think really big.”
Like your data, it’s time to react in real time and learn why data streaming is essential to your data strategy.
This blog explores how cloud service providers (CSPs) and managed service providers (MSPs) increasingly recognize the advantages of leveraging Confluent to deliver fully managed Kafka services to their clients. Confluent enables these service providers to deliver higher value offerings to wider...
With Confluent sitting at the core of their data infrastructure, Atomic Tessellator provides a powerful platform for molecular research backed by computational methods, focusing on catalyst discovery. Read on to learn how data streaming plays a central role in their technology.