Simplifying Asynchronous Service Communication with Kafka, Debezium and the Outbox Pattern

Loggi is one of the Brazilian unicorns which is transforming logistics through technology to connect the entire country. At Loggi, we've widely used the CDC pattern with Debezium + Kafka to capture the changes from our main database as a way of communication between services, as an Event Notification pattern. Later, we concluded this wasn't enough, since we often need to join information from multiple tables to take an action. In this talk, we describe this journey, and how we leverage Debezium’s Outbox pattern to transactionally emit events following the Event-Carried State Transfer pattern, where all the needed information is sent to Kafka at once, therefore avoiding consumer requests to enrich the event data. Additionally, we describe an Event Broker service we’ve implemented to transform Kafka’s consumption from pull-based to push-based, helping other teams to integrate their services into this new architecture faster.