While Apache Kafka® is the central nervous system for streaming data, Kafka Connect is the highway that allows traffic to flow in and out of Kafka without the need to write a producer/consumer. The question that arises here is how do we fit more cars (records) on this highway and get better throughput?
In this presentation, we will use JDBC source and sink connectors as examples of how to tune source/sink connectors:
- Elaborate internals of source vs sink connectors(client/producer vs consumer/client)
- List of configurations to tune for producer/consumer
- List of configurations to tune from source/sink JDBC connector
- Throughput results of tuned vs untuned
- Which metrics to monitor
- What can’t be tuned (data conversion between JDBC client and producer/consumer)
Let us Connect all of our systems through the largest possible highway!
Presenter
Catalin Pop
ConfluentStarted off my career as an IBM Software Engineer where I eventually ended up being a Solution Architect/Lead Software Engineer on two different projects. After almost 2.5 years, I decided to join a new startup(Confluent) to learn all about data streaming. This is where I first started getting my exposure to Kafka and Connect. I'm currently the SME for Kafka Connect within the organization and work closely with our PMs/Engineer Leads to help improve the product.