[Webinar + Demo] BMW Group’s Omnichannel Transformation Using Data Streaming | Join!
A well-known debate: tabs or spaces? Sure, we could set up a Google Form to collect this data, but where’s the fun in that? Let’s settle the debate, Kafka-style. We’ll use the new confluent-kafka-javascript client (not in general availability yet) to build an app that produces the current state of the vote counts to a Kafka topic and consumes from that same topic to surface them to a JavaScript frontend.
Why are we using this client in particular? It comes from Confluent and is intended for use with Apache Kafka® and Confluent Platform. It’s compatible with Confluent’s cloud offering as well. It builds on concepts from the two most popular Kafka JavaScript client libraries: KafkaJS and node-rdkafka. The functionality is based on node-rdkafka, however, it also provides a way to interface with the library via methods similar to those in KafkaJS due to their developer-friendy nature. There are two APIs: the first implements the functionality based on node-rdkafka; the second is a promisified API with the methods akin to those in KafkaJS. By choosing this client, we can access wide functionality and have a smooth developer experience via the dev-friendly methods.
If you’re migrating from either of these two APIs, you can find the migration guide here.
We use a version of the API that is convertible from node-rdkafka on the producing side. All of the code in this section is written in producer.js. You’ll get an opportunity to see how we can use the promisified API when we create the code for consuming the events.
First, you need to get set up in Confluent Cloud. There are detailed steps in the GitHub repository README which holds the code in its entirety.
Here’s the basic shape of one of our Kafka messages:
It’s truncated (there are seven questions total) but it gives you the idea. Each question object holds the number of votes for each option, as well as a “lastClicked” boolean to keep track of the interface state.
Now, we’re not using stream processing this time around (we’ll leave that as an exercise for you, dear reader), instead we update the state in the backend here, and consume the last message in the topic each time before changing it to get the last message. We want to be able to conditionally start from the first offset, or perhaps in the future, and re-consume the last committed message if we have to restart, so here we use manual commits.
Here you can see that by using the API which avails itself to methods similar to those in rd-kafka, we can obtain the offset [1] via the consumer.committed
method. Then, we [2] check to see if it’s an invalid offset. If so, we restart at the beginning. Otherwise, we assign the consumer its topic, partition, and offset and run consumer.consume()
to start the consumption process.
Here you can see that we’re [2] manually committing as well as updating a decoded
variable [1].
It’s that variable that we will update and produce to our topic in our producer code:
First [1], we retrieve the vote and count from the user interface, then [2] we update the count. Next [3], we clear the lastClicked
state before [4] updating it for our specific vote. Last [5], we produce the message containing the updated state to the topic.
[1] This logic is called in an express route, which will pass information from the user interface to the producer via the [2] request body.
For the consumer side, we will use the promisified API. Important: this is not the same API the example uses in the above section.
If you’re migrating from node-rdkafka, you import like so:
However, if you’re using the promisified API, like in the following code snippets we’re about to highlight, you import Kafka in the following manner:
This code, along with its context, lies in the file consumer.js.
We [1] connect to the consumer, [2] subscribe to the topic, [3] run the consumer and manage each message coming in from the topic, [4] find the last clicked message, and [5] extract the question ID and vote count. [6] These pieces of information are emitted to our websocket.
The code above demonstrates a very important piece of logic for consumer management. We [1] call the disconnect method when exiting the terminal. The code within the disconnect
method ensures that the consumer [2] commits the offset from the message in the topic before [3] disconnecting. That way the consumer can properly pick up from where it left off.
How do we get these events to surface to the user interface in a readable manner? For the frontend, we use mostly vanilla JavaScript and CSS. That way, if someone wants to use this project as an example but they’re using a frontend framework in JavaScript, they can replicate the logic easily no matter what framework they’re using.
We use a bit of jQuery to post the vote information to our route which we saw in producer.js
:
On a user’s click, this sends the [1] vote object, the [2] question ID (gleaned from the interface’s element ID), and the [3] timestamp to the send-to-kafka-topic
route for the Kafka producer to send to the topic. The timestamp is present should we ever want to revisit this project and, say, perform some windowing on the data stream.
Here in the frontend, we write some logic that triggers when the [1] socket receives an event. [2] Then we create a variable for the question_id, and [3] retrieve the DOM element corresponding to that id. [4] We retrieve the label for that bar as well, which we'll [a] update in the following chain of logic. Then, we check if the [5] left-hand option has more votes, [6] if there's been a tie, or if in the remaining case, [7], the right-hand side has more votes. [a] We then update the DOM accordingly.
Side note: you might wonder where it’s all deployed. We deployed the frontend on Amplify, and the backend (the consumer.js
and producer.js
files) on EC2. They’re connected via a load balancer.
Where can you go from here? There are lots of opportunities to grow this project. For example, you could implement a visualization: X amount of devs who voted for tabs also voted for spaces. Or you could fingerprint the user to prevent them from cheating (don’t worry, we haven’t done this, we trust you not to vote multiple times). A third possibility is to create a mechanism for when a user wants to change their vote.
If you’re interested in learning more about Kafka or confluent-kafka-javascript, we recommend these resources:
The Kafka 101 course
A basic introduction to confluent-kafka-javascript
Kafka client language guides
The Confluent YouTube channel
Previously, I posted about the Kafka REST Proxy from Confluent, which provides easy access to a Kafka cluster from any language. That post focused on the motivation, low-level examples, and […]
Apache Kafka® is an event streaming platform used by more than 30% of the Fortune 500 today. There are numerous features of Kafka that make it the de-facto standard for […]