Webscale Workflow Engine With Kafka

Have you ever wanted a reliable way to run and visualize hundreds of thousands of workflows of various shapes and sizes every second? At a smaller scale, did you simply want to consolidate on a single infrastructure to run workflows defined in different formats?

In this talk, we introduce a workflow engine concept that only uses Kafka to persist state transitions and execution results. The system banks on Kafka’s high reliability, transactionality, and high scale to keep setup and operating costs low. Our first target use case is our Continuous Integration (CI) systems. We’ll present a demo of the system in action with parallel and nested CI workflow definitions of varying declaration formats. You’ll see real-time visualization of progress with the help of Kafka. We’ll then cause some chaos and load generation to showcase how retries and scale ups work. We’ll conclude by walking through the architecture, explain how extension points work, and contrasting it to other popular workflow engines.

Our goal is to convince the community that Kafka is a viable choice as the persistence layer for problems where you have to do state machine processing.