Kafka in the Cloud: Why it’s 10x better with Confluent | Find out more
Kafka Connect is quintessential for most production-grade Kafka data pipelines. As a core Kafka Connect team supporting more than 75 connectors at Confluent, we needed a platform that not only enables easy repeatability of tests across a breadth of connectors but also facilitates benchmarking and extensive analysis of diverse test runs. This session goes over the challenges in designing a framework to test connector performance.
We will cover specific aspects of Connector testing, fine tuning parameters like connector configurations, clusters specifications for optimal performance. The framework integrates with various tools such as Jmeter, Hammerdb, ChaosMesh, Prometheus etc., scaled for Cloud-native environments. We will further go over how we optimized connector performance for maximum public record poll rates, cpu/memory usage, throughput, latencies for different data formats.
Attendees learn in detail how real-world scenarios can be simulated in a test setup for a few connectors like S3 Sink Connector. We also go over some generic guidelines for performance testing of Kafka Connectors and practical considerations for implementing such test frameworks in general.