Technology

Apache Kafka has a data structure called the “request purgatory”. The purgatory holds any request that hasn’t yet met its criteria to succeed but also hasn’t yet resulted in an […]

The amount of work that got done by the community in the last month is truly impressive, especially considering how many conferences took place in September. Let’s take a look at the highlights: The […]

I ran into the schema-management problem while working with my second Hadoop customer. Until then, there was one true database and the database was responsible for managing schemas and pretty […]

I am very excited that LinkedIn’s deployment of Apache Kafka has surpassed 1.1 trillion (yes, trillion with a “t”, and 4 commas) messages per day. This is the largest deployment of Apache […]

September is the start of the fall conference season. Between Strata + Hadoop World New York and ApacheCon: Big Data Europe, there is plenty to keep us busy learning.

This post was jointly written by Neha Narkhede, original co-creator of Apache Kafka, and Flavio Junqueira, co-creator of Apache ZooKeeper. Many distributed systems that we build and use currently rely on dependencies like […]

Welcome to the first edition of Log Compaction, a monthly digest of highlights in the Apache Kafka and stream processing community. Today’s edition are the highlights from July and early […]

One of the things I realised while doing research for my book is that contemporary software engineering still has a lot to learn from the 1970s. As we’re in such […]

Apache Kafka is widely used to enable a number of data intensive operations from collecting log data for analysis to acting as a storage layer for large scale real-time stream […]

Previously, I posted about the Kafka REST Proxy from Confluent, which provides easy access to a Kafka cluster from any language. That post focused on the motivation, low-level examples, and […]

This post has been written in collaboration with Derrick Harris from Mesosphere and Joe Stein, a Kafka committer. For an updated version of this article, please see Apache Mesos, Apache Kafka and […]

Building operational simplicity into distributed systems, especially for nuanced behaviors, is somewhat of an art and often best achieved after gathering production experience. Apache Kafka‘s popularity can be attributed in […]

This is a repost of a recent article that I wrote for ODBMS. In the last few years, there has been significant growth in the adoption of Apache Kafka. Current […]

This is an edited transcript of a talk I gave at the Craft Conference 2015. The video and slides are also available.