Oh no! Yet another Kafka operator for Kubernetes

Yes, we opensourced yet another Apache Kafka operator for Kubernetes. This might seem bizarre, considering the alternatives that are already available (they exist but there are not too many), so you may be wondering, ‘Why?’ Well, keep reading and we’ll tell you: from design gaps and features we believe are necessary to operate Kafka on K8s, through my personal fix for Envoy, to some of our specific usage scenarios.tl;dr So to return to our initial question, ‘Why?’We could not find another operator which really matched our customer usage patterns when running large Kafka clusters on-prem, and in multi and hybrid-cloud environments.The nearest we got to finding such an operator (that was open source) was written in Java, and we prefer working in Golang (we wanted to contribute back the features we needed).We believe the other operators out there have some fundamental design flaws, which we were not willing to accommodate.We opensourced an Apache Kafka operator built in Golang.The Pipeline platform builds on top of the operator and allows us to run and operate managed Kafka as a service across all supported platforms.As a side effect of working on some complex operators like the Kafka, Istio or Vault, we opensourced a Kubernetes objectmatcher library to help avoid unnecessary Kubernetes object updates.Ok, that just about does it for complaints and excuses; let’s move on to some real technical content. If you’d like to see the operator in action make sure you try out the Pipeline Platform, the easiest way to run Kafka on Kubernetes deployed to multiple clouds or on-prem with the Kafka Spotguide - Zookeeper, pre-configured Prometheus monitoring, Kafka security, centralized log collection, external accees and lots more are out of the box services.Overview Apache Kafka is an open-source distributed streaming platform, and some of the main features of the Kafka-operator are:the provisioning of secure and production ready Kafka clustersfine grained broker configuration supportadvanced and highly configurable External Access via LoadBalancers using Envoygraceful Kafka cluster scaling and rebalancingmonitoring via Prometheusencrypted communication using SSLautomatic reaction and self healing based on alerts (plugin system, with meaningful default alert plugins) using Cruise ControlMotivation At Banzai Cloud we are building a Kubernetes distribution, PKE, and a hybrid-cloud container management platform, Pipeline, that operate Kafka clusters (among other types) for our customers. Apache Kafka predates Kubernetes and was designed mostly for static on-premise environments. State management, node identity, failover, etc all come part and parcel with Kafka, so making it work properly on Kubernetes and on an underlying dynamic environment can be a challenge.There are already several approaches to operating Kafka on Kubernetes, however, we did not find them appropriate for use in a highly dynamic environment, nor capable of meeting our customers’ needs. At the same time, there is substantial interest within the Kafka community for a solution which enables Kafka on Kubernetes, both in the open source and closed source space.Helm chartYaml filesStrimzi Kafka OperatorConfluent operatorJoin us as we take a deep dive into some of the details of the most popular pre-existing solutions, as well as our own:Banzai CloudKrallisticStrimziConfluentOpen sourceApache 2Apache 2Apache 2NoMaintainedYesNo (discontinued)YesN/AFine grained broker config supportYes (learn more)Limited via StatefulSetLimited via StatefulSetLimited via StatefulSetFine grained broker volume supportYes (learn more)Limited via StatefulSetLimited via StatefulSetLimited via StatefulSetMonitoringYesYesYesYesEncryption using SSLYesYesYesYesRolling UpdateYesNoNoYesCluster external accessesEnvoy (single LB)NodeportNodeport or LB/brokerYes (N/A)User Management via CRDYesNoYesNoTopic management via CRDYesNoYesNoReacting to AlertsYes (Prometheus + Cruise ControlNoNoNoGraceful Cluster Scaling (up and down)Yes (using Cruise Control)NoNoYes-if you find any of this information inaccurate, please let us know, and we’ll fix itKafka is a stateful application. The first piece of the puzzle is the Broker, which is a simple server capable of creating/forming a cluster with other Brokers. Every Broker has his own unique configuration which differs slightly from all others - the most relevant of which is the unique broker ID.All Kafka on Kubernetes operators use StatefulSet to create a Kafka Cluster. Just to quickly recap from the K8s docs:StatefulSet manages the deployment and scaling of a set of Pods, and provide guarantees about their ordering and uniqueness. Like a Deployment, a StatefulSet manages Pods that are based on an identical container spec. Unlike a Deployment, a StatefulSet maintains sticky identities for each of its Pods. These pods are created from the same spec, but are not interchangeable: each has a persistent identifier that is maintained across any rescheduling.How does this looks from the perspective of Apache Kafka?With StatefulSet we get:unique Broker IDs generated during Pod startupnetworking between brokers with headless servicesunique Persistent Volumes for BrokersUsing StatefulSet we lose:the ability to modify the configuration of unique Brokersto remove a specific Broker from a cluster (StatefulSet always removes the most recently created Broker)to use multiple, different Persistent Volumes for each BrokerThe Banzai Cloud Kafka Operator uses simple Pods, ConfigMaps, and PersistentVolumeClaims, instead of StatefulSet. Using these resources allows us to build an Operator which is better suited to Kafka.With the Banzai Cloud Kafka operator we can:modify the configuration of unique Brokersremove specific Brokers from clustersuse multiple Persistent Volumes for each BrokerFeatures To highlight some of the features we needed and were not possible with the operators available, please keep reading (I know, I know, this post is getting long).Fine Grained Broker Config Support We needed to be able to react to events in a fine-grained way for each Broker - and not in the limited way StatefulSet does (which, for example, removes the most recently created Brokers). Some of the available solutions try to overcome these deficits by placing scripts inside the container to generate configs at runtime, whereas the Banzai Cloud Kafka operator’s configurations are deterministically placed in specific Configmaps.Graceful Kafka Cluster Scaling Here at Banzai Cloud, we know how to operate Kafka at scale (we are contributors and have been operating Kafka on Kubernetes for years now). We believe, however, that LinkedIn has even more experience than we do. To scale Kafka clusters both up and down gracefully, we integrated LinkedIn’s Cruise-Control to do the hard work for us. We already have good defaults (i.e. plugins) that react to events, but we also allow our users to write their own.External Access via LoadBalancer The Banzai Cloud Kafka operator externalizes access to Kafka using a dynamically (re)configured Envoy proxy. Using Envoy allows us to use a single LoadBalancer, so there’s no need for a LoadBalancer for each Broker.Communication via SSL The operator fully automates Kafka’s SSL support. Users must provide and install the right certificate as a Kubernetes Secret, however, the Pipeline platform is capable of automating this process, as well.Monitoring via Prometheus The Kafka operator exposes Cruise-Control and Kafka JMX metrics to Prometheus.Reacting on Alerts The Kafka Operator acts as a Prometheus Alert Manager. It receives alerts defined in Prometheus, and creates actions based on Prometheus alert annotations.Currently, there are three default actions (which can be extended):upscale cluster (add a new Broker)downscale cluster (remove a Broker)add additional disk to a BrokerScenarios As highlighted in the features section, we removed the reliance on StatefulSet, we dig deep and write bunch of code for the operator to allow scenarios as the following ones (this is by far not a complete list, get in touch if you have a requirement, question or would like to talk about the operator).Vertical capacity scaling We’ve encountered many situations in which the horizontal scaling of a cluster is impossible. When only one Broker is throttling and needs more CPU or requires additional disks (because it handles the most partitions), a StatefulSet-based solution is useless, since it does not distinguishes between replicas’ specifications. The handling of such a case requires unique Broker configurations. If we need to add a new disk to a unique Broker, with a StatefulSet-based solution, we waste a lot of disk space (and money). Since it can’t add a disk to a specific Broker, the StatefulSet adds one to each replica.With the Banzai Cloud Kafka operator, adding a new disk to any Broker is as easy as changing a CR configuration. Similarly, any Broker-specific configuration can be done on a Broker by Broker basis.An unhandled error with Broker #1 in a three Broker cluster In the event of an error with Broker #1, we want to handle it without disrupting the other Brokers. Maybe we would like to temporarily remove this Broker from the cluster, and fix its state, reconciling the node that serves the node, or maybe reconfigure the Broker using a new configuration. Again, when using StatefulSet, we lose the ability to remove specific Brokers from the cluster. StatefulSet only supports a field name replica that determines how many replicas an application should use. If there’s a downscale/removal, this number can be lowered, however, this means that Kubernetes will remove the most recently added Pod (Broker #3) from the cluster - which, in this case, happens to suit our purposes quite well. To remove the #1 Broker from the cluster, we need to lower the number of brokers in the cluster from three to one. This will cause a state in which only one Broker is live, while we kill the brokers that handle traffic; the Banzai Cloud Kafka operator supports removing specific brokers without disrupting traffic in the cluster.Fine grained Broker config support Apache Kafka is a stateful application, where Brokers create/form a cluster with other Brokers. Every Broker is uniquely configurable (we support heterogenous environments, in which no nodes are the same, act the same or have the same specifications - from the infrastructure up through the Brokers’ Envoy configuration). Kafka has lots of Broker configs, which can be used to fine tune specific brokers, and we did not want to limit these to ALL Brokers in a StatefulSet. We support unique Broker configs.In each of the three scenarios lister above, we decided to not use StatefulSet in our Kafka Operator, relying, instead, on Pods, PVCs and ConfigMaps. We believe StatefulSet is very convenient starting point, as it handles roughly 80% of scenarios but introduces huge limitations when running Kafka on Kubernetes in production.Monitoring based control Use of monitoring is essential for any application, and all relevant information about Kafka should be published to a monitoring solution. When using Kubernetes, the de facto solution is Prometheus, which supports configuring alerts based on previously consumed metrics. We wanted to build a standards-based solution (Prometheus and Alert Manager) that could handle and react to alerts automatically, so human operators wouldn’t have to. The Banzai Cloud Kafka operator supports alert-based Kafka cluster management.LinkedIn’s Cruise Control We have a lot of experience in operating both Kafka and Kubernetes at scale. However, we believe that LinkedIn knows how to operate Kafka even better than we do. They built a tool, called Cruise Control, to operate their Kafka infrastructure, and we wanted to build an operator which handled the infrastructure but did not reinvent the wheel insofar as operating Kafka. We didn’t want to redevelop proven concepts, but wanted to create an operator which leveraged our deep Kubernetes expertise (after all, we’ve already built a CNCF certified Kubernetes distribution, PKE and a hybrid cloud container management platform, Pipeline) by handling all Kafka infrastructure related issues in the way we thought best. We believe managing Kafka is a separate issue, for which there already exist some unique tools and solutions that are standard across the industry, so we took LinkedIn’s Cruise Control and integrated it with the operator.This post has grown pretty long, pretty quickly, so I’ll wrap things up here. Give our operator a try and let us know what you think, and please star the Kafka GitHub repo. Should you need help, or if you have any questions, get in touch with us by joining our #kafka-operator channel on Slack.Banzai Cloud changing how private clouds get built to simplify the development, deployment, and scaling of complex applications, bringing the full power of Kubernetes and Cloud Native technologies to developers and enterprises everywhere. #multicloud #hybridcloud #BanzaiCloud, visit ​banzaicloud.com​