# kafka-operator

**Repository Path**: mirrors_adobe/kafka-operator

## Basic Information

- **Project Name**: kafka-operator
- **Description**: Oh no! Yet another Kafka operator for Kubernetes
- **Primary Language**: Unknown
- **License**: Apache-2.0
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2020-09-24
- **Last Updated**: 2026-09-05

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

<p align="center">

![Koperator](https://img.shields.io/github/v/release/adobe/koperator?label=Koperator)
![Released](https://img.shields.io/github/release-date/adobe/koperator?label=Released)
![License](https://img.shields.io/github/license/adobe/koperator?label=License)
![Go version (latest release)](https://img.shields.io/github/go-mod/go-version/adobe/koperator/0.30.0)

</p>

---

<p align="center">

![Go version](https://img.shields.io/github/go-mod/go-version/adobe/koperator/master)
[![Go Report Card](https://goreportcard.com/badge/github.com/adobe/koperator)](https://goreportcard.com/report/github.com/adobe/koperator)
![CI](https://img.shields.io/github/actions/workflow/status/adobe/koperator/ci.yml?branch=master&label=CI)
![CI](https://img.shields.io/github/actions/workflow/status/adobe/koperator/codeql-analysis.yml?branch=master&label=CodeQL)
![Image](https://img.shields.io/github/actions/workflow/status/adobe/koperator/e2e-test.yaml?branch=master&label=E2E)
![Helm chart](https://img.shields.io/github/actions/workflow/status/adobe/koperator/helm.yml?branch=master&label=Helm%20chart)


</p>

# Koperator

Koperator is an open-source operator that automates the provisioning, management, and autoscaling of Apache Kafka clusters on Kubernetes.
Unlike other solutions that rely on StatefulSets, Koperator has been built with a unique architecture that provides greater flexibility and functionality for managing Apache Kafka. This architecture allows for fine-grained configuration and management of individual brokers.

Some of the main features of Koperator are:

- the provisioning of secure and production-ready Kafka clusters
- fine-grained broker-by-broker configuration support
- advanced and highly configurable external access
- graceful Kafka cluster scaling and rebalancing
- detailed Prometheus metrics
- encrypted communication using SSL
- automatic reaction and self-healing based on alerts using [Cruise Control](https://github.com/linkedin/cruise-control)
- graceful rolling upgrades
- advanced topic and user management via Kubernetes Custom Resources
- Cruise Control task management via Kubernetes Custom Resources

## AI Code Wiki

Google indexes this repo at:
https://codewiki.google/github.com/adobe/koperator

## Architecture

Kafka is a stateful application, and the Kafka Broker is a server that can create and form a cluster with other Brokers. Each Broker has its own unique configuration, the most important of which is the unique broker ID.

Most Kubernetes operators that manage Kafka rely on [`StatefulSets`](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/) to create a Kafka Cluster.
While StatefulSets provide unique Broker IDs generated during Pod startup, networking between brokers with headless services, and unique Persistent Volumes for Brokers, they have a few restrictions. For example, Broker configurations cannot be modified independently, and a specific Broker cannot be removed from the cluster - a StatefulSet always removes the most recently created Broker. Furthermore, multiple, different Persistent Volumes cannot be used for each Broker.

*Koperator takes a different approach by using simple `Pods`, `ConfigMaps`, and `PersistentVolumeClaims` instead of `StatefulSets`. These resources allow us to build an Operator that is better suited to manage Apache Kafka.*
With Koperator, you can modify the configuration of unique Brokers, remove specific Brokers from clusters, and use multiple Persistent Volumes for each Broker.

If you want to learn more about our design motivations and the scenarios that drove us to create Koperator, please continue reading on our [documentation page](https://opensource.adobe.com/koperator/docs/scenarios/).

![Koperator architecture](docs/img/kafka-operator-arch.png)

## Quick start

This quick start guide will walk you through the process of deploying Koperator on an existing Kubernetes cluster and provisioning a Kafka cluster using its custom resources.

### Prerequisites

To complete this guide, you will need a Kubernetes cluster (with a suggested minimum of 6 vCPUs and 8 GB RAM). You can run the cluster locally using Kind or Minikube.

> The quick start will help you set up a functioning Kafka cluster on Kubernetes. However, it does not include guidance on the installation of Prometheus and cert-manager, which are necessary for some of the more advanced functionality.

#### Install ZooKeeper

The version of Kafka that is installed by the operator requires Apache ZooKeeper. You'll need to deploy a ZooKeeper cluster if you don’t already have one.

1. Install ZooKeeper using [Pravega’s Zookeeper Operator](https://github.com/pravega/zookeeper-operator).

```sh
helm install zookeeper-operator --repo https://charts.pravega.io zookeeper-operator --namespace=zookeeper --create-namespace
```

1. Create a ZooKeeper cluster.

```sh
kubectl create -f - <<EOF
apiVersion: zookeeper.pravega.io/v1beta1
kind: ZookeeperCluster
metadata:
    name: zookeeper-server
    namespace: zookeeper
spec:
    replicas: 1
    image:
        repository: ghcr.io/adobe/zookeeper-operator/zookeeper
        tag: 3.8.4-0.2.15-adobe-20260423
    persistence:
        reclaimPolicy: Delete
EOF
```

1. Verify that ZooKeeper has been deployed.

```sh
> kubectl get pods -n zookeeper

NAME                                         READY   STATUS    RESTARTS   AGE
zookeeper-server-0                           1/1     Running   0          27m
zookeeper-operator-54444dbd9d-2tccj          1/1     Running   0          28m
```

### Install Koperator

You can deploy Koperator using a Helm chart from GitHub Container Registry (OCI). Complete the following steps.

1. Install the Koperator `CustomResourceDefinition` resources (adjust the version number to the Koperator release you want to install). This is performed in a separate step to allow you to uninstall and reinstall Koperator without deleting your already installed custom resources.

```sh
kubectl apply -f https://raw.githubusercontent.com/adobe/koperator/refs/heads/master/config/base/crds/kafka.banzaicloud.io_cruisecontroloperations.yaml
kubectl apply -f https://raw.githubusercontent.com/adobe/koperator/refs/heads/master/config/base/crds/kafka.banzaicloud.io_kafkaclusters.yaml
kubectl apply -f https://raw.githubusercontent.com/adobe/koperator/refs/heads/master/config/base/crds/kafka.banzaicloud.io_kafkatopics.yaml
kubectl apply -f https://raw.githubusercontent.com/adobe/koperator/refs/heads/master/config/base/crds/kafka.banzaicloud.io_kafkausers.yaml
```

2. Install Koperator into the `kafka` namespace using the OCI Helm chart from GitHub Container Registry. Use `--skip-crds` since the CRDs were already installed in the previous step - without it, Helm's own CRD install can conflict with the `kubectl apply` above ([#265](https://github.com/adobe/koperator/issues/265)):

> 📦 **View available versions**: [ghcr.io/adobe/helm-charts/kafka-operator](https://github.com/adobe/koperator/pkgs/container/helm-charts%2Fkafka-operator/versions)

OCI registries have no floating "latest" tag, so `--version` is required (replace with your desired version, see available versions above):

```sh
helm install kafka-operator oci://ghcr.io/adobe/helm-charts/kafka-operator --version 0.30.0 --namespace=kafka --create-namespace --skip-crds
```

#### Pull and inspect the chart before installation

```sh
# Pull the chart locally
helm pull oci://ghcr.io/adobe/helm-charts/kafka-operator --version 0.30.0

# Extract and inspect
tar -xzf kafka-operator-0.30.0.tgz
helm template kafka-operator ./kafka-operator/

# Install from local chart
helm install kafka-operator ./kafka-operator/ --namespace=kafka --create-namespace --skip-crds
```

1. Create the Kafka cluster using the `KafkaCluster` custom resource. The quick start uses a minimal custom resource, but there are other examples in the same directory.

```sh
kubectl create -n kafka -f https://raw.githubusercontent.com/adobe/koperator/master/config/samples/simplekafkacluster.yaml
```

1. Verify that the Kafka cluster has been created.

```sh
> kubectl get pods -n kafka

kafka-0-nvx8c                             1/1     Running   0          16m
kafka-1-swps9                             1/1     Running   0          15m
kafka-2-lppzr                             1/1     Running   0          15m
kafka-cruisecontrol-fb659b84b-7cwpn       1/1     Running   0          15m
kafka-operator-operator-8bb75c7fb-7w4lh   2/2     Running   0          17m
```

### Test Kafka cluster

To test the Kafka cluster let's create a topic and send some messages.

1. You can use the `KafkaTopic` CR to create a topic called `my-topic`:

```sh
kubectl create -n kafka -f - <<EOF
apiVersion: kafka.banzaicloud.io/v1alpha1
kind: KafkaTopic
metadata:
    name: my-topic
spec:
    clusterRef:
        name: kafka
    name: my-topic
    partitions: 1
    replicationFactor: 1
    config:
        "retention.ms": "604800000"
        "cleanup.policy": "delete"
EOF
```

1. If SSL encryption is disabled for Kafka, you can use the following commands to send and receive messages within a Kubernetes cluster.

To send messages, run this command and type your test messages:

```sh
kubectl -n kafka run kafka-producer -it --image=adobe/kafka:2.13-3.9.1 --rm=true --restart=Never -- /opt/kafka/bin/kafka-console-producer.sh --bootstrap-server kafka-headless:29092 --topic my-topic
```

To receive messages, run the following command:

```sh
kubectl -n kafka run kafka-consumer -it --image=adobe/kafka:2.13-3.9.1 --rm=true --restart=Never -- /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server kafka-headless:29092 --topic my-topic --from-beginning
```

## Documentation

For detailed documentation on the Koperator project, see the [Koperator documentation website](https://opensource.adobe.com/koperator/).

## Issues and contributions

We use GitHub to track issues and accept contributions.
If you would like to raise an issue or open a pull request, please refer to our [contribution guide](./CONTRIBUTING.md).

If you use Koperator in a production environment, we encourage you to add yourself to the list of production [adopters](https://github.com/adobe/koperator/blob/master/ADOPTERS.md).

## Community

Engage with the Adobe OSS maintainers of this repo via Github issues in this repo.

[![Contributors](https://contrib.rocks/image?repo=adobe/koperator&max=30)](https://github.com/adobe/koperator/graphs/contributors)

## License

Copyright (c) 2023 [Cisco Systems, Inc.](https://www.cisco.com) and/or its affiliates
Copyright (c) 2025 Adobe. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

[http://www.apache.org/licenses/LICENSE-2.0](http://www.apache.org/licenses/LICENSE-2.0)

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

## Trademarks

Apache Kafka, Kafka, and the Kafka logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries.