Robinhood’s Kafkaproxy: Decoupling Kafka Consumer Logic from Application Business Logic

Apache Kafka is Robinhood’s most mission-critical infrastructure. It is used in every line of Robinhood’s business, from stock and crypto trading to self-clearing, market data and data science. Most of our microservices are written in either Golang or Python, and we have multiple librdkafka-based client library wrappers (kafkahood) developed in both languages. As Robinhood grows, it has become challenging for infrastructure engineers to manage the different requirements that the multitude of application teams have for their producers and consumers. To efficiently manage all Kafka consumers at Robinhood, we have developed a Consumer Proxy that decouples the following:

● Kafka consumer logic from message processing in application business logic

● Resource utilization in Kafka consumer from that in application container

● Kafka consumer failure from application failure

The Consumer Proxy is built in a Kubernetes sidecar container, it offers stateless processing, and it manages commits and timeouts. Most importantly, applications need the ability to scale up during market hours and scale down after extended trading hours, and the Consumer Proxy will allow them to easily rebalance consumer groups during these scaling events. Eventually, specific language library wrappers can remain a """"thin"""" layer whose complexities are only to facilitate the business logic of the corresponding application team with what they should do with the payload.

Unifying Kafka consumers and producers for an application team in one sidecar proxy container allows the infrastructure team to effectively manage clients at Robinhood. If your organization is looking to centralize Kafka consumption logic to a singular client library (instead of multiple different client libraries), please attend this talk to see how Robinhood does it so that the infrastructure team can focus development on a singular library rather than N different client libraries.