[Webinar] Build Your GenAI Stack with Confluent and AWS | Register Now

Presentation

OpenLineage for Stream Processing

« Kafka Summit London 2024

OpenLineage is an open platform for the collection and analysis of data lineage, which includes an open standard for lineage data collection, integration libraries for the most common tools, and a metadata repository/reference implementation (Marquez).

In recent months, stream processing, which is an important use case for Apache Kafka, has gained the particular focus of the OpenLineage community with many useful features completed or begun, including:

  • A seamless OpenLineage & Apache Flink integration,
  • Support for streaming jobs in Marquez,
  • Progress on a built-in lineage API within the Flink codebase.

Cross-platform lineage allows for a holistic overview of data flow and its dependencies within organizations, including stream processing.

This talk will provide an overview of the most recent developments in the OpenLineage Flink integration and share what’s in store for this important collaboration.

This talk is a must-attend for those wishing to stay up-to-date on lineage developments in the stream processing world.

Related Links

How Confluent Completes Apache Kafka eBook

Leverage a cloud-native service 10x better than Apache Kafka

Confluent Developer Center

Spend less on Kafka with Confluent, come see how