Implementing End-To-End Tracing

Can you answer how a given event came to be? Is it an aggregation, a combination of multiple events with different sources? What are its origins?

Given the growing complexity of event streaming architectures - stateful processing, joins, fan-outs, multi-cluster flows - it is increasingly important to be able to accurately answer those questions, understand data flows and capture data provenance.

This talk will walk through how to use and extend OpenTelemetry Java agent auto instrumentation to achieve full end-to-end traceability in Kafka event streaming architectures involving multi-cluster deployments, the Connect platform, stateful KStream applications and ksqlDB workloads.

We will cover:

• Distributed Tracing concepts - context propagation and the OpenTelemetry implementation stack;
• Java agent auto instrumentation, problems faced when instrumenting service platforms (Connect and ksqlDB), stateful applications (KStreams and ksqlDB) and how auto instrumentation can be extended using loadable extensions to solve those problems;
• Demo of an end-to-end tracing implementation and a highlight of the interesting use cases it enables.