Pinterest’s Journey to a Automated, Efficient, and Low-Maintenance Kafka Platform

At Pinterest, Apache® Kafka runs on ~2000 brokers across 50+ clusters, moving data at petabyte scale every day. Through investing in automation and focusing on cost efficiency, we have seen positive results in dev velocity, scalability, stability, and cost footprint, enabling everyone to quickly and easily launch Kafka-supported use cases that power business growth, while keeping our platform low-maintenance.

We believe that sharing the innovations along our journey to a highly-automated, efficient, and low-maintenance Kafka platform can provide immense value for industry partners. This talk will cover the following topics in-depth, bringing inspiration and actionable takeaways for each attendee who might face the same problems operating Kafka at scale:

1. Automation: >80% reduction in overall onboarding time for Kafka-supported applications

• Automated endpoint discovery with PubSub Client (PSC), available standalone and as Flink connector (open-sourced)
• Automated resource provisioning for Kafka-based applications via drag-and-drop UI, reducing setup time from 30 mins to 5 mins
• Automated broker & cluster management and recovery via Orion (open-sourced)
• Automated client-side error handling via PSC (open-sourced)

2. Efficiency: >25% reduction in platform cost

• Reducing cluster sizes and optimizing efficiency via using heterogeneous hardware
• Reducing broker footprint via storage tiering
• Reducing cross-region KafkaMirror resource utilization via Shallow Mirror (KIP-712)

3. Low-Maintenance: simplifying business processes

• Generic CDC framework via Debezium / Kafka Connect
• Improving cost visibility and attribution for multi-tenant environment, enabling cost-driven decision making
• Moving towards service tiering to further simplify customer onboarding and platform operations