With ksqlDB, the event streaming database, developers will be able to build event streaming applications that use stream processing with less hassle and complexity
Mountain View, Calif. – November 20, 2019 – Confluent, Inc., the event streaming platform pioneer, today announced ksqlDB, an event streaming database that seeks to unify the multiple systems involved in stream processing into a single, easy-to-use solution for building event streaming applications. ksqlDB aims to bring the approachable feel of relational databases to the world of stream processing, making it easier for developers to build the event streaming applications that are increasingly required to compete and succeed in the modern era.
People demand immediacy in every aspect of their digital lives. These expectations were set by companies who harnessed the power of real-time by shifting their thinking from “data” to “events.” While data gives information about the way things are, events provide the richer story about how things got there. And by streaming events in real-time, innovative companies have transformed how we shop, bank, and hail rides. It is these experiences that are shaping what people expect from every other organization they interact with – seventy-three percent of people say that one extraordinary experience with a company raises their expectations of others.
However, building event streaming applications is complex, and requires knowledge of multiple distributed systems that have to be integrated, secured, monitored, and operated as one. This complexity paired with the surging demand for event streaming applications has pushed Apache Kafka® to becoming the second-highest paid tech skill in the United States, according to a report by Dice.com. As a result, building event streaming applications is out of reach for most organizations.
“Event streaming applications are quickly becoming a foundational application development paradigm” said Neha Narkhede, co-founder and chief product officer at Confluent. “But the complexity required to build them has left great ideas unexecuted. ksqlDB is a major evolution of KSQL to make it easy for any organization to harness event streams in Apache Kafka. With ksqlDB, we’re reducing the number of pieces developers needed to understand and put together to write event streaming applications. By using just Kafka and ksqlDB, any organization can leverage event streaming data to compete in the modern world.”
Introducing ksqlDB
ksqlDB is an event streaming database purpose-built to help developers create event streaming applications that use stream processing. It represents a major evolution of KSQL, Confluent’s streaming SQL engine for real time data processing. With the addition of traditional database-like lookups in the form of pull queries and embedded connectors capable of running directly inside its servers, ksqlDB represents a new category of infrastructure — an event streaming database. With ksqlDB, the complex application architecture required to build event streaming applications is reduced to two components: Kafka and ksqlDB. The operational complexity required to build event streaming applications is drastically reduced, allowing developers and enterprises to build real-time applications without requiring significant budget or time investments.
Pull Queries
Many event streaming applications require subscriptions to events when they happen (these queries “push” updates to the app) and point in time lookups of state (that the app “pulls” from). In traditional architectures, push and pull queries were accessed from multiple disparate systems, adding to the complexity of event streaming apps. But with pull queries, Confluent is simplifying the development of event streaming applications by integrating traditional database-like lookups on top of materialized tables built for stored data. By adding pull queries, ksqlDB offers one SQL query language to work with both push and pull queries in one model.
Take a ride sharing app for example. Because drivers are always moving, the location of your driver on your map needs to be continuously updated. This is accomplished through push queries that the app subscribes to updated to. On the other hand, information like the price of your ride is locked in once and doesn’t change during your trip. This is information that the app pulls from a data store. In traditional event streaming applications, supporting both push and pull queries meant working with two different data systems. But with ksqlDB, they are unified in one platform, simplifying the architecture.
Connector Management
Previously, building event streaming applications required running a dedicated Kafka Connect cluster to move data between Kafka and external data systems like databases, key-value stores, search indexes, and file systems. This added to the complexity of building event streaming applications. ksqlDB solves this problem with built-in support for a large range of pre-built connectors capable of running directly inside its servers, rather than relying on a separate solution to get event data from and to every external data system. Now developers can directly control and execute connectors built to work with Kafka Connect, making it easier to both source and sink event data with external systems, creating a simplified event streaming architecture.
ksqlDB is available today under the Confluent Community license.
“I think of event stream processing as the infrastructure for ‘systems of opportunity’ that identify, react and respond immediately while data is still in motion. Until recently, businesses required tip-of-the-spear teams to build and manage these strategic systems,” according to Maureen Fleming, Program Vice President, IDC. “With improvements in ease of development and ongoing management, building out systems of opportunity will become core to business innovation for all types and sizes of organizations.”
Additional Resources:
About Confluent
Confluent, founded by the original creators of Apache Kafka®, pioneered the enterprise-ready event streaming platform. With Confluent, organizations benefit from the first event streaming platform built for the enterprise with the ease of use, scalability, security and flexibility required by the most discerning global companies to run their business in real time. Companies leading their respective industries have realized success with this new platform paradigm to transform their architectures to streaming from batch processing, spanning on-premises and multi-cloud environments. Backed by Benchmark, Index Ventures and Sequoia Capital, Confluent is headquartered in Palo Alto and London with offices globally. To learn more, please visit www.confluent.io. Download Confluent Platform and Confluent Cloud at www.confluent.io/download.
Confluent and associated marks are trademarks or registered trademarks of Confluent, Inc.
Apache® and Apache Kafka® are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. No endorsement by the Apache Software Foundation is implied by the use of these marks. All other trademarks are the property of their respective owners.