Building an Interactive Query Service in Kafka Streams

Kafka Streams is a powerful stream processing library that offers stateful operations. Along with the stateful operations is a unique feature of Kafka Streams, Interactive Queries, or IQ. IQ allows you to leverage the state of the application from the outside by directly querying the state stores.

But Kafka Streams is a distributed application; often, no single instance has a state store with all the data. So Kafka Streams provides the infrastructure, so a developer doesn't need to worry about querying the correct instance.

But the implementation of communication between app instances (Remote Procedure Call or RPC) is not provided, leaving the developer to Google searches on how to get started building one. In this talk, I'll discuss and demonstrate what's needed to build an RPC mechanism between Kafka Stream instances, including:

• The background of Interactive Queries
• Using Spring Boot to expose your Interactive Query Service
• How to route queries between app instances.
• Building a view to render the results

You'll leave with the knowledge on how to get started building an Interactive Query service and a reference application to get started.