How the New Confluent Hub Makes Finding Connectors Easier Than Ever

Confluent Hub launched in 2018 as a place to discover and share Apache Kafka^® and Confluent Platform plugins. Users have found the site a much better place to discover useful code than seeking out individual git repos. These plugins include connectors, Single Message Transforms (SMTs), and converters for Kafka Connect. At launch, we had a humble inventory of about a dozen plugins. Now, there are over 140 available, including 100+ that are fully supported by Confluent and our partners!

Since its launch, we’ve looked for ways to improve Confluent Hub, and today we’re happy to introduce a complete frontend redesign, based on user feedback. In particular, we’ve tried to make it much easier to parse plugins across dimensions that matter most to developers.

Finding what you need

The most notable update to Confluent Hub is the addition of a detailed plugin taxonomy that lets you filter options across a number of key plugin dimensions.

Plugin type

Determining the plugin type you need is likely your first concern as a developer. The plugin types on Confluent Hub are as follows:

• Sink connector: sink connectors consume messages from Kafka topics and export them to external systems (e.g., Amazon S3 or MySQL).
• Source connector: source connectors consume data from external systems (e.g., Amazon S3 or MySQL) and write them to Kafka topics.
• Transform: SMTs are applied to messages as they flow through Connect. SMTs transform inbound messages after a source connector has produced them but before they are written to Kafka. Alternatively, SMTs transform outbound messages before they are sent to a sink connector.
• Converter: converters change the format of data from bytes to a Connect internal data format, and vice versa. They are necessary in order for a Kafka Connect deployment to support a particular data format when writing to or reading from Kafka.

Enterprise support

Depending on where you’re doing your development, you may or may not need access to enterprise-level support. You might even be quite adventurous with your own projects. Lots of companies we talk to, however, aren’t comfortable putting something into production without the ability to get expert support on the phone. To help both types of users navigate the available plugins, Confluent Hub lets you filter by enterprise support type:

• Confluent supported: Confluent enterprise support for these plugins is included as part of a Confluent Platform subscription. (You can contact us to learn more.)
• Partner supported: enterprise support is available from the partner who maintains the plugin.
• None: Many open source plugins fall in this category. No enterprise support is available, although varying levels of community support may be available. An open source Git repo and the Confluent Community Slack are often good places to look.

Verification

Similar to enterprise support, you may or may not be developing in an environment that requires everything to be ready for “enterprise scale” from day zero. It’s impossible to point to a single benchmark that quantifies this, but these verification levels are our attempt to identify at least those plugins that are known to be built for high performance and stability. These categories include:

• Confluent Built: built, tested, and supported by Confluent.
• Confluent Tested: community plugins, which are tested and supported by Confluent.
• Verified Gold: partner built and supported integrations that meet the highest level of functional compatibility with the Confluent Platform ecosystem. (See the Confluent Verified Integration FAQs for further details.)
• Verified Standard: partner built and supported integrations that meet a standard level of functional compatibility with the Confluent Platform ecosystem. (See the Confluent Verified Integration FAQs for further details.)
• None: community contributions that have not been verified or tested by Confluent. This doesn’t mean they’re not good—we just haven’t had a chance to rigorously test them.

License

This one is pretty straightforward:

• Commercial: requires a Confluent Platform or partner subscription. Confluent supported plugins can be used for free on a single broker cluster (Developer License) or free for 30 days on an unlimited number of brokers (Evaluation License).
• Free: available for free through an open source or community license.

Available in Confluent Cloud

Our fully managed Apache Kafka as a service product, Confluent Cloud, offers some of these connectors, fully managed, on leading public clouds like Google Cloud Platform (GCP), Microsoft Azure, and Amazon Web Services (AWS).

Let us know what you think or submit a plugin

That’s a quick overview of what’s changed in the Confluent Hub frontend. We’re always looking for ways to improve the product and serve the community better, and one of the key ways we do this is by adding new open source and partner connectors to Confluent Hub. If you would like to get your code in front of more developers, or if you’d just like to give some feedback, send us an email at confluent-hub@confluent.io. We look forward to hearing from you!