Apache Kafka
Data Products, Data Contracts, and Change Data Capture
Change data capture is a popular method to connect database tables to data streams, but it comes with drawbacks. The next evolution of the CDC pattern, first-class data products, provide resilient pipelines that support both real-time and batch processing while isolating upstream systems...
Introducing Confluent Cloud Freight Clusters
Learn how the latest innovations in Kora enable us to introduce new Confluent Cloud Freight clusters, which can save you up to 90% at GBps+ scale. Confluent Cloud Freight clusters are now available in Early Access.
Contributing to Apache Kafka®: How to Write a KIP
Learn how to contribute to open source Apache Kafka by writing Kafka Improvement Proposals (KIPs) that solve problems and add features! Read on for real examples.
Apache Kafka, Samza, and the Unix Philosophy of Distributed Data
One of the things I realised while doing research for my book is that contemporary software engineering still has a lot to learn from the 1970s. As we’re in such […]
Compression in Apache Kafka is now 34% faster
Apache Kafka is widely used to enable a number of data intensive operations from collecting log data for analysis to acting as a storage layer for large scale real-time stream […]
Making Apache Kafka Elastic With Apache Mesos
This post has been written in collaboration with Derrick Harris from Mesosphere and Joe Stein, a Kafka committer. For an updated version of this article, please see Apache Mesos, Apache Kafka and […]
Hands-Free Kafka Replication: A Lesson in Operational Simplicity
Building operational simplicity into distributed systems, especially for nuanced behaviors, is somewhat of an art and often best achieved after gathering production experience. Apache Kafka‘s popularity can be attributed in […]
Using logs to build a solid data infrastructure (or: why dual writes are a bad idea)
This is an edited transcript of a talk I gave at the Craft Conference 2015. The video and slides are also available.
Compatibility Testing For Apache Kafka
Testing is one of the hardest parts of building reliable distributed systems. Kafka has long had a set of system tests that cover distributed operation but this is an area […]
Bottled Water: Real-time integration of PostgreSQL and Kafka
Summary: Confluent is starting to explore the integration of databases with event streams. As part of the first step in this exploration, Martin Kleppmann has made a new open source […]
A Comprehensive REST Proxy for Kafka
As part of Confluent Platform 1.0 released about a month ago, we included a new Kafka REST Proxy to allow more flexibility for developers and to significantly broaden the number […]
Apache Kafka 0.8.2.1 release
The Apache Kafka community just announced the 0.8.2.1 release. This is a a bug fix release and fixes 4 critical issues reported in the 0.8.2.0 release (the full list of […]
How to Choose the Number of Topics/Partitions in a Kafka Cluster?
Note For the latest, check out the blog posts Apache Kafka® Made Simple: A First Glimpse of a Kafka Without ZooKeeper and Apache Kafka Supports 200K Partitions Per Cluster.
Turning the database inside-out with Apache Samza
This is an edited and expanded transcript of a talk I gave at Strange Loop 2014. The video recording (embedded below) has been watched over 8,000 times. For those of […]