Log Compaction – Highlights in the Apache Kafka^® and Stream Processing Community – June 2017

We are very excited for the GA for Kafka release 0.11.0.0 which is just days away. This release is bringing many new features as described in the previous Log Compaction blog post.

The most notable new feature is Exactly Once Semantics (EOS).  Kafka’s EOS capabilities provide more stringent idempotent producer semantics with exactly once, in-order delivery per partition, and stronger transactional guarantees with atomic writes across multiple partitions. Together, these strong semantics make writing applications easier and expand Kafka’s addressable use cases. You can learn more about EOS in the online talk on June 29, 2017.

The recent 2017 Apache Kafka Report showed that 37% of Kafka adopters are now using the Kafka Connect API. As part of the streaming ETL paradigm, Kafka Connectors make it very simple to load data from other systems into Kafka and to extract Kafka data into other systems. The full list of Kafka Connectors has recently been updated with some new ones contributed by the community including:

• AWS S3 sink connector
• Redis source connector
• RabbitMQ source connector
• GitHub source connector
• Amazon Kinesis source connector
• MongoDB sink connector

Video recordings and slide decks from the sessions at Kafka Summit NYC are now available. And in a few days, the Kafka Summit committee will announce the list of sessions for the upcoming Kafka Summit in San Francisco. If you would like to attend the conference which is happening on August 28, 2017, please register soon.

Register for Kafka Summit SF

Notable blogs and presentations:

• Scrapping Your Yearlong Project: a CTO’s description of how the Kafka stream pipeline architecture compares to other approaches to data flow
• How Pandora Uses Kafka: Pandora transitioned its ad tracking system from a batch-oriented system into a real-time one, and adopted Kafka as a core component of its new stream processing system
• Scale-out for Complex Event Processing: horizontal scale-out architecture built around Kafka
• Real-Time Stream Processing for Internet of Things: stream processing enables quick processing of large amounts of data from multiple IOT sources, in real-time
• Configuring Kafka Connect to log REST HTTP messages to a separate file: helping developers work with log files in Kafka Connect

If you want to directly engage with the Kafka community, you can engage in a variety of ways: Google Group, Slack, Reddit, LinkedIn, Twitter, and please join us at a Kafka Meetup group in your area! Alternatively, please feel free to reach out to us at community@confluent.io