Building a fully Kafka-based product as a Data Scientist

A data analytics project for a food processing factory revealed that business problems could be solved and processes improved by implementing streaming applications.

BAADER's Transport Manager consists of a few microservices based on Kafka Streams and several ksqlDB queries running on a managed ksqlDB in Confluent Cloud. It tracks poultry trailers to improve animal welfare. Moreover, additional data are associated, such as weather information and ETA, to optimize for on-time delivery.

This session guides through the development process from a Data Scientist's perspective having a limited software development skillset. A closer look is taken on challenges being tackled such as creating and testing topologies, acquiring knowledge about state stores, streams and KTables, and dealing with ksqlDB breaking changes. Takeaways are presented about great developer resources for Kafka Streams and handy ksqlDB functions, to help developers with similar skills working with Apache Kafka.