Apache Kafka®️ 비용 절감 방법 및 최적의 비용 설계 안내 웨비나 | 자세히 알아보려면 지금 등록하세요

Dawn of Kafka DevOps: Managing Kafka Clusters at Scale with Confluent Control Center

작성자:

When managing Apache Kafka® clusters at scale, tasks that are simple on small clusters turn into significant burdens. To be fair, a lot of things turn into significant burdens at scale, and it’s Confluent Control Center’s job to ease as many of them as possible. In Confluent Platform 5.2, Control Center has grown a couple of new features that make large deployments a little more pleasant to manage: It has become much better at managing configuration changes among a large number of brokers, and it scales to a larger number of managed partitions. Let us explain these two in a bit of detail.

Dynamic broker configuration

Two challenges present themselves when configuring a large number of brokers: visualizing the differences in broker configuration and modifying them without resorting to rolling restarts everywhere. These are problems endemic to any distributed system that uses a large number of the same node type, and they become more costly as the number of nodes grows. Confluent Platform 5.2 makes solving both of these problems a bit easier.

In previous versions of Control Center, you could view and download broker configurations, which was good as far as it went. If you wanted to compare broker configurations (hardly an unusual thing to do when trying to troubleshoot a misbehaving cluster), you were left to do the diffing on your own. As of 5.2, you can now directly view the differences in configurations from within Control Center.

Confluent Control Center

Relatedly, KIP-226 enabled dynamic broker reconfiguration since Apache Kafka 1.1. Put this together with the configuration view, and you now have a powerful way to get misconfigured brokers whipped into shape. Not only can you view their configurations side by side, but you can also make changes to parameters as needed without having to restart the brokers, which is an enormous time savings no matter how you manage your deployment. Of course, not every broker configuration parameter can be changed dynamically. See the documentation (or, if you please, the Apache Kafka wiki) for a complete list of which parameters this applies to.

Confluent Control Center

Dynamic broker configuration is enabled by default. To disable, set confluent.controlcenter.broker.config.edit.enable=false in the control-center.properties. And remember, you can always consult the documentation for more information.

Improved scalability

Control Center gives you visibility into the behavior of each topic that it manages, which means it has to keep track of the state of each partition in each topic. (No, this is not a terribly surprising claim of software architecture, but work with me here.) This, in turn, implies that Control Center has its own upper bounds on how large a deployment it can manage. Just how many topics Control Center can handle turns out to be somewhat of a complicated question, depending on replication factor and the distribution of topic partition counts. A cluster may have many topics with a small number of partitions, a small number of topics with many partitions or something in between; and a cluster may have a large replication factor or a small one.

Those statistical variations notwithstanding, Control Center has leveled up significantly in Confluent Platform 5.2. It can now comfortably handle around 120,000 individual partition replicas. Assuming a very typical replication factor of three, this means it can now manage around 40,000 individual partitions. If you take a typical topic partition count of six, this translates to a cluster of about 6,700 topics—which, you might be thinking, is a lot. Most users of Control Center don’t manage deployments that large; in fact, our data tells us that this covers more than 90% of existing customer workloads.

Cluster with Partitions

Operating large systems is always a lot of work, and Confluent Platform is singularly focused on making it easier for you. Confluent Platform 5.2 has moved the ball forward in two areas: Control Center’s ability to manage larger clusters, and its ability to help you more easily compare broker configurations and dynamically change them. If you’re running a big cluster, these features will make life that much more pleasant for you.

If you’re still not using Control Center or other parts of the Confluent Platform, you know what to do! Go here, download and check it out.

Other articles in this series:

  • Tim은 Confluent의 개발자 관계 부문 부사장으로서, 팀원과 함께 모든 개발자가 스트리밍 데이터와 새로운 도구 세트에 액세스할 수 있도록 노력하고 있습니다. 컨퍼런스에서 정기적으로 연사로 활동하며 YouTube에서 복잡한 기술 주제를 알기 쉽게 설명합니다. 현재 미국 캘리포니아주 마운틴뷰에서 아내, 의붓딸과 함께 살고 있습니다. 슬하에 장성한 자녀 셋, 의붓자식 셋, 손주 넷이 있습니다.

  • Viktor Gamov는 Apache Kafka를 기반으로 이벤트 스트리밍 플랫폼을 개발하는 Confluent에서 근무하며 개발자를 지지합니다. 컨설턴트 시절에는 오픈 소스 기술을 사용해 엔터프라이즈 애플리케이션 아키텍처를 구축하기 위한 포괄적인 전문 지식을 쌓았습니다. 그는 아키텍트와 개발자들이 지연 시간이 짧은 확장형 고가용성 분산 시스템을 설계하고 개발하도록 돕습니다. 분산 시스템, 스트리밍 데이터, JVM, DevOps 분야의 전문 컨퍼런스 연사로서 JavaOne, Devoxx, OSCON, QCon 같은 행사에서 정기적으로 강연을 합니다. O'Reilly에서 출간한 Enterprise Web Development를 공동 저술했으며 Confluent 블로그에도 글을 쓰고 있습니다.

이 블로그 게시물이 마음에 드셨나요? 지금 공유해 주세요.