Kafka makes it possible to ingest and process large amounts of data concurrently by decoupling your data streams. This successful approach introduces some challenges when monitoring Kafka pipelines - to understand what’s happening, we need to monitor every single component and how they interact with each other.
In this talk, we will take a close look at Kafka’s architecture as well as the key infrastructure, JVM, and system metrics you should monitor for each of its components. Then, we will walk through how to diagnose common Kafka performance anomalies through observing patterns in the metrics for the various components. Finally, we will walk through setting up an open source observability pipeline with OpenTelemetry to enable you to collect and process Kafka metrics at scale.
Presenter
Daniel Kim
New RelicDaniel Kim is a Senior Developer Relations Engineer at New Relic and the founder of Bit Project, a 501(c)(3) nonprofit dedicated make tech accessible to underserved communities. He wants to inspire generations of students in tech to be the best they can be through inclusive, accessible developer education. He is passionate about diversity & inclusion in tech, good food, and dad jokes.
Presenter
Antón Rodríguez
New RelicI have focused my career on crafting developer platforms, demonstrating expertise in various areas. I embarked by mastering orchestrators, progressing to API Gateways, and ultimately delving into Data Streaming over the past few years. These pursuits have enabled me to amass valuable insights into distributed systems, messaging protocols, and Cloud infrastructure. While I enjoy engaging with diverse technologies, my deepest proficiency lies in Kafka, Flink, and Java, where I have honed my skills and gained extensive experience.