On Track with Apache Kafka: Building a Streaming ETL solution with Rail Data

« Kafka Summit Europe 2021

Do you want to know what streaming ETL actually looks like in practice? Or what you can REALLY do with Apache Kafka once you get going—using config & SQL alone?

This project integrates live data from the UK rail network via ActiveMQ along with data from other sources to build a fully-functioning platform. It includes analytics through Elasticsearch and exploring graph relationships in Neo4j, as well as real-time alerts delivered through Telegram.

This talk will show how I built the system, and include live demos and code samples of the salient integration points in ksqlDB and Kafka Connect.

The data may be domain-specific but the challenges of handling batch and stream data to drive both applications and analytics are encountered by many. This talk will give people lots of concrete examples of patterns and techniques for integration and stream processing.

Presenter

Robin Moffatt

Decodable

Robin is a Principal DevEx Engineer at Decodable as well as an Oracle Groundbreaker Ambassador. His career has always involved data, from the old worlds of COBOL and DB2, through the worlds of Oracle and Hadoop, and into the current world with Kafka. His particular interests are analytics, systems architecture, performance testing and optimization. He blogs at http://cnfl.io/rmoff and http://rmoff.net/ and can be found tweeting grumpy geek thoughts as @rmoff. Outside of work he enjoys drinking good beer and eating fried breakfasts, although generally not at the same time.

On Track with Apache Kafka: Building a Streaming ETL solution with Rail Data

Presenter

Robin Moffatt

Related Links

How Confluent Completes Apache Kafka eBook

Leverage a cloud-native service 10x better than Apache Kafka

Confluent Developer Center

Spend less on Kafka with Confluent, come see how