[Webinar] Build Your GenAI Stack with Confluent and AWS | Register Now
Log Compaction – Highlights in the Apache Kafka® and Stream Processing Community – June 2017
We are very excited for the GA for Kafka release 0.11.0.0 which is just days away. This release is bringing many new features as described in the previous Log Compaction blog post.
The most notable new feature is Exactly Once Semantics (EOS). Kafka’s EOS capabilities provide more stringent idempotent producer semantics with exactly once, in-order delivery per partition, and stronger transactional guarantees with atomic writes across multiple partitions. Together, these strong semantics make writing applications easier and expand Kafka’s addressable use cases. You can learn more about EOS in the online talk on June 29, 2017.
The recent 2017 Apache Kafka Report showed that 37% of Kafka adopters are now using the Kafka Connect API. As part of the streaming ETL paradigm, Kafka Connectors make it very simple to load data from other systems into Kafka and to extract Kafka data into other systems. The full list of Kafka Connectors has recently been updated with some new ones contributed by the community including:
- AWS S3 sink connector
- Redis source connector
- RabbitMQ source connector
- GitHub source connector
- Amazon Kinesis source connector
- MongoDB sink connector
Video recordings and slide decks from the sessions at Kafka Summit NYC are now available. And in a few days, the Kafka Summit committee will announce the list of sessions for the upcoming Kafka Summit in San Francisco. If you would like to attend the conference which is happening on August 28, 2017, please register soon.
Notable blogs and presentations:
- Scrapping Your Yearlong Project: a CTO’s description of how the Kafka stream pipeline architecture compares to other approaches to data flow
- How Pandora Uses Kafka: Pandora transitioned its ad tracking system from a batch-oriented system into a real-time one, and adopted Kafka as a core component of its new stream processing system
- Scale-out for Complex Event Processing: horizontal scale-out architecture built around Kafka
- Real-Time Stream Processing for Internet of Things: stream processing enables quick processing of large amounts of data from multiple IOT sources, in real-time
- Configuring Kafka Connect to log REST HTTP messages to a separate file: helping developers work with log files in Kafka Connect
If you want to directly engage with the Kafka community, you can engage in a variety of ways: Google Group, Slack, Reddit, LinkedIn, Twitter, and please join us at a Kafka Meetup group in your area! Alternatively, please feel free to reach out to us at community@confluent.io
Did you like this blog post? Share it now
Subscribe to the Confluent blog
Introducing Apache Kafka® 3.9
We are proud to announce the release of Apache Kafka 3.9.0. This is a major release, the final one in the 3.x line. This will also be the final major release to feature the deprecated Apache ZooKeeper® mode. Starting in 4.0 and later, Kafka will always run without ZooKeeper.
Preparing the Consumer Fetch: Kafka Producer and Consumer Internals, Part 3
In this third installment of a blog series examining Kafka Producer and Consumer Internals, we switch our attention to Kafka consumer clients, examining how consumers interact with brokers, coordinate their partitions, and send requests to read data from Kafka topics.