Transactions in Action: the Story of Exactly Once in Apache Kafka

« Current 2023

Transactions were added to Apache Kafka with KIP-98. While much of the protocol remains intact, transactions in Kafka have evolved over time to handle edge cases and errors found over the years. KIP-890 hopes to cover most of the remaining gaps in the protocol. This talk will give a refresher on transactions and idempotency and chronicle the various KIPs that improved the protocol over the years. We will also discuss the problem of hanging transactions and how KIP-890 hopes to solve it as well as strengthen the transactional protocol overall.

Presenter

Justine Olshan

Confluent

Justine graduated from Carnegie Mellon University in 2020 with a degree in computer science. During summer 2019, she was a software engineering intern at Confluent where she worked on improving the Apache Kafka® producer. After graduating, she returned full time to Confluent and continues to work on improving Kafka through various KIPs including KIP-516 which introduced topic IDs to Kafka and KIP-890 which strengthened the transactional protocol. She became an Apache Kafka committer in 2022 and PMC member in 2023.

Transactions in Action: the Story of Exactly Once in Apache Kafka

Presenter

Justine Olshan

Related Links

How Confluent Completes Apache Kafka eBook

Leverage a cloud-native service 10x better than Apache Kafka

Confluent Developer Center

Spend less on Kafka with Confluent, come see how