Level Up Your Kafka Skills in Just 5 Days | Join Season of Streaming On-Demand
Data privacy is a crucial issue in today's world: from medical to finance, from retail to the public sector, data must be handled in accordance with national laws, corporate policies and industry best-practices. Being able to ensure data is safe at all times is now a requirement in many use cases.
We’ll start by looking at the threat models that Apache Kafka users often have and requirements such as end-to-end and record encryption. We’ll cover why Apache Kafka’s built-in security features, such as authentication and wire-level encryption, don’t address them. We’ll then look at the various solutions we investigated, weighing their architectural pros and cons.
We’ll detail the solution we ended up building, which is an entirely open source end-to-end encryption mechanism using a L7 proxy for Apache Kafka. We’ll describe in detail some of the key concepts of our implementation as well as some pitfalls we hit, so attendees can learn to safeguard their data’s confidentiality, integrity and availability.