Versioned State Stores in Kafka Streams

KIP-889 is the first in a sequence of KIPs to introduce versioned key-value stores into Kafka Streams. Versioned key-value stores enhance stateful processing capabilities by allowing users to store multiple record versions per key, rather than only the single latest version per key as is the case for existing key-value stores today. Storing multiple record versions per key unlocks use cases such as true temporal stream-table joins: when an out-of-order record arrives on the stream-side, Kafka Streams can produce the correct join result by looking ""back in time"" for the table state at the timestamp of the stream-side record. Foreign-key joins will see similar benefits, and users can also support custom use cases in their applications by running interactive queries to look up older record versions from versioned state stores, or by using them in custom processors.

This talk will introduce versioned state stores starting from the basics, discuss the stream-table join use case as motivation, operational considerations for users who'd like to use them, briefly touch on implementation in doing so, and also cover the timeline for when various pieces of functionality can be expected. By May 2023, KIP-889 will be complete and follow-up KIPs will be opened/in-progress.