In this talk Admiral Foobar describes all the traps, gotchas and challenges of his (eventually successful) fight to replace the built-in RocksDB and In Memory State Stores with an external, hosted database. In his intergalactic quest for instant rebalances and restores, he describes how to
- navigate an exactly-once-semantics asteroid minefield with epoch machinery
- subvert zombie writers by building a fencing mechanism
- dominate the Global KTables with truly shared state
- respect the tombstones of your fallen comrade events in all situations
At the end of this talk, you should feel ready to join the fight against the RocksDB Empire and build your own state store.
Presenter
Sophie Blee-Goldman
ResponsiveSophie is a committer and member of the PMC on Apache Kafka, with a focus on Kafka Streams. She is a founding engineer at Responsive where she is building a managed, cloud-native, Kafka Streams offering.
Presenter
Almog Gavra
ResponsiveAlmog Gavra is a Co-Founder at Responsive helping build ksqlDB. His introduction to stream processing was at LinkedIn, where he worked on various parts of the search infrastructure including the real-time index updates.