代码拉取完成，页面将自动刷新
Apache Kafka is a distributed streaming platform used for building real-time data pipelines and streaming apps. It is horizontally scalable, fault-tolerant, wicked fast, and runs in production in thousands of companies. Kafka requires a connection to a Zookeeper service.
The main folder of this repository contains a functional docker-compose.yml file. Run the application using it as shown below:
$ curl -sSL https://raw.githubusercontent.com/bitnami/bitnami-docker-kafka/master/docker-compose.yml > docker-compose.yml
$ docker-compose up -d
DOCKER_CONTENT_TRUST=1 to verify the integrity of the images.This CVE scan report contains a security report with all open CVEs. To get the list of actionable security issues, find the "latest" tag, click the vulnerability report link under the corresponding "Security scan" field and then select the "Only show fixable" filter on the next page.
Deploying Bitnami applications as Helm Charts is the easiest way to get started with our applications on Kubernetes. Read more about the installation in the Bitnami Apache Kafka Chart GitHub repository.
Bitnami containers can be used with Kubeapps for deployment and management of Helm Charts in clusters.
Non-root container images add an extra layer of security and are generally recommended for production environments. However, because they run as a non-root user, privileged tasks are typically off-limits. Learn more about non-root containers in our docs.
Dockerfile linksNOTE: Debian 8 images have been deprecated in favor of Debian 9 images. Bitnami will not longer publish new Docker images based on Debian 8.
Learn more about the Bitnami tagging policy and the difference between rolling tags and immutable tags in our documentation page.
2-ol-7, 2.3.1-ol-7-r20 (2/ol-7/Dockerfile)2-debian-9, 2.3.1-debian-9-r13, 2, 2.3.1, 2.3.1-r13, latest (2/debian-9/Dockerfile)Subscribe to project updates by watching the bitnami/kafka GitHub repo.
The recommended way to get the Bitnami Kafka Docker Image is to pull the prebuilt image from the Docker Hub Registry.
docker pull bitnami/kafka:latest
To use a specific version, you can pull a versioned tag. You can view the list of available versions in the Docker Hub Registry.
docker pull bitnami/kafka:[TAG]
If you wish, you can also build the image yourself.
docker build -t bitnami/kafka:latest 'https://github.com/bitnami/bitnami-docker-kafka.git#master:2/debian-9'
If you remove the container all your data and configurations will be lost, and the next time you run the image the database will be reinitialized. To avoid this loss of data, you should mount a volume that will persist even after the container is removed.
Note! If you have already started using your database, follow the steps on backing up and restoring to pull the data from your running container down to your host.
The image exposes a volume at /bitnami/kafka for the Kafka data. For persistence you can mount a directory at this location from your host. If the mounted directory is empty, it will be initialized on the first run.
Using Docker Compose:
This requires a minor change to the docker-compose.yml file present in this repository:
kafka:
...
volumes:
- /path/to/kafka-persistence:/bitnami/kafka
...
NOTE: As this is a non-root container, the mounted files and directories must have the proper permissions for the UID
1001.
Using Docker container networking, a Kafka server running inside a container can easily be accessed by your application containers.
Containers attached to the same network can communicate with each other using the container name as the hostname.
In this example, we will create a Kafka client instance that will connect to the server instance that is running on the same docker network as the client.
$ docker network create app-tier --driver bridge
Use the --network app-tier argument to the docker run command to attach the Zookeeper container to the app-tier network.
$ docker run -d --name zookeeper-server \
--network app-tier \
-e ALLOW_ANONYMOUS_LOGIN=yes \
bitnami/zookeeper:latest
Use the --network app-tier argument to the docker run command to attach the Kafka container to the app-tier network.
$ docker run -d --name kafka-server \
--network app-tier \
-e ALLOW_PLAINTEXT_LISTENER=yes \
-e KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper-server:2181 \
bitnami/kafka:latest
Finally we create a new container instance to launch the Kafka client and connect to the server created in the previous step:
$ docker run -it --rm \
--network app-tier \
-e KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper-server:2181 \
bitnami/kafka:latest kafka-topics.sh --list --zookeeper zookeeper-server:2181
When not specified, Docker Compose automatically sets up a new network and attaches all deployed services to that network. However, we will explicitly define a new bridge network named app-tier. In this example we assume that you want to connect to the Kafka server from your own custom application image which is identified in the following snippet by the service name myapp.
version: '2'
networks:
app-tier:
driver: bridge
services:
zookeeper:
image: 'bitnami/zookeeper:latest'
networks:
- app-tier
kafka:
image: 'bitnami/kafka:latest'
networks:
- app-tier
myapp:
image: 'YOUR_APPLICATION_IMAGE'
networks:
- app-tier
IMPORTANT:
- Please update the
YOUR_APPLICATION_IMAGEplaceholder in the above snippet with your application image- Configure Kafka and ZooKeeper persistence, and configure them either via environment variables or by mounting configuration files.
- In your application container, use the hostname
kafkato connect to the Kafka server
Launch the containers using:
$ docker-compose up -d
The configuration can easily be setup with the Bitnami Kafka Docker image using the following environment variables:
ALLOW_PLAINTEXT_LISTENER: Allow to use the PLAINTEXT listener. Default: noKAFKA_INTER_BROKER_USER: Kafka inter broker communication user. Default: admin. Default: adminKAFKA_INTER_BROKER_PASSWORD: Kafka inter broker communication password. Default: bitnamiKAFKA_BROKER_USER: Kafka client user. Default: userKAFKA_BROKER_PASSWORD: Kafka client user password. Default: bitnamiKAFKA_ZOOKEEPER_USER: Kafka Zookeeper user. No defaultsKAFKA_ZOOKEEPER_PASSWORD: Kafka Zookeeper user password. No defaultsKAFKA_CERTIFICATE_PASSWORD: Password for certificates. No defaults.KAFKA_HEAP_OPTS: Kafka's Java Heap size. Default: -Xmx1024m -Xms1024mAdditionally, any environment variable beginning with KAFKA_CFG_ will be mapped to its corresponding Kafka key. For example, use KAFKA_CFG_BACKGROUND_THREADS in order to set background.threads.
docker run --name kafka -e KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181 -e ALLOW_PLAINTEXT_LISTENER=yes bitnami/kafka:latest
or by modifying the docker-compose.yml file present in this repository:
kafka:
...
environment:
- KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
...
In order to use internal and external clients to access Kafka brokers you need to configure one listener for each kind of clients.
To do so, add the following environment variables to your docker-compose:
environment:
- KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
- ALLOW_PLAINTEXT_LISTENER=yes
+ - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
+ - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,PLAINTEXT_HOST://:29092
+ - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
And expose the extra port:
ports:
- '9092:9092'
+ - '29092:29092'
These clients will use the docker hostname to connect to Kafka.
kafka-console-producer.sh --broker-list kafka:9092 --topic test
kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic test --from-beginning
These clients will use the localhost listener created in the port 29092 to connect to Kafka.
kafka-console-producer.sh --broker-list localhost:29092 --topic test
kafka-console-consumer.sh --bootstrap-server localhost:29092 --topic test --from-beginning
More info about Kafka listeners can be found in this great article
The Bitnami Kafka docker image disables the PLAINTEXT listener for security reasons. You can enable the PLAINTEXT listener by adding the next environment variable, but remember that this configuration is not recommended for production.
ALLOW_PLAINTEXT_LISTENER=yes
In order to configure SASL authentication over SSL, you should define the proper listener by passing the following env vars:
KAFKA_CFG_LISTENERS=SASL_SSL://:9092
KAFKA_CFG_ADVERTISED_LISTENERS=SASL_SSL://:9092
You must also use your own certificates for SSL. You can drop your Java Key Stores files into /opt/bitnami/kafka/conf/certs.
If the JKS is password protected (recommended), you will need to provide it to get access to the keystores:
KAFKA_CERTIFICATE_PASSWORD=myCertificatePassword
The following script can help you with the creation of the JKS and certificates:
Keep in mind the following notes:
kafka.example.com. After entering this value, when prompted "What is your first and last name?", enter this value as well.The following docker-compose file is an example showing how to mount your JKS certificates protected by the password certificatePassword123.
Additionally it is specifying the Kafka container hostname and the credentials for the broker, inter-broker and zookeeper users.
version: '2'
services:
zookeeper:
image: 'bitnami/zookeeper:latest'
ports:
- '2181:2181'
environment:
- ZOO_ENABLE_AUTH=yes
- ZOO_SERVER_USERS=kafka
- ZOO_SERVER_PASSWORDS=kafka_password
kafka:
image: 'bitnami/kafka:latest'
hostname: kafka.example.com
ports:
- '9092'
environment:
- KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
- KAFKA_CFG_LISTENERS=SASL_SSL://:9092
- KAFKA_CFG_ADVERTISED_LISTENERS=SASL_SSL://:9092
- KAFKA_ZOOKEEPER_USER=kafka
- KAFKA_ZOOKEEPER_PASSWORD=kafka_password
- KAFKA_INTER_BROKER_USER=interuser
- KAFKA_INTER_BROKER_PASSWORD=interpassword
- KAFKA_BROKER_USER=user
- KAFKA_BROKER_PASSWORD=password
- KAFKA_CERTIFICATE_PASSWORD=certificatePassword123
volumes:
- './kafka.keystore.jks:/opt/bitnami/kafka/conf/certs/kafka.keystore.jks:ro'
- './kafka.truststore.jks:/opt/bitnami/kafka/conf/certs/kafka.truststore.jks:ro'
By default, communications that happens between brokers are authenticated. You can provide your own credentials using this environment variables:
KAFKA_INTER_BROKER_USER: Kafka inter broker communication user. Default: adminKAFKA_INTER_BROKER_PASSWORD: Kafka inter broker communication password. Default: bitnamiBy default, any Kafka client needs to authenticate before can connect to a broker. You can provide your own credentials using this environment variables:
KAFKA_BROKER_USER: Kafka client user. Default: userKAFKA_BROKER_PASSWORD: Kafka client user password. Default: bitnamiIn order to authenticate Kafka against a Zookeeper server with SASL authentication you should provide the next environment variables:
KAFKA_ZOOKEEPER_USER: Kafka Zookeeper user. No defaults.KAFKA_ZOOKEEPER_PASSWORD: Kafka Zookeeper user password. No defaults.Below you can see a complete Docker Compose example:
version: '2'
services:
zookeeper:
image: 'bitnami/zookeeper:latest'
ports:
- '2181:2181'
environment:
- ZOO_ENABLE_AUTH=yes
- ZOO_SERVER_USERS=kafka
- ZOO_SERVER_PASSWORDS=kafka_password
kafka:
image: 'bitnami/kafka:latest'
ports:
- '9092'
environment:
- KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
- KAFKA_CFG_LISTENERS=SASL_SSL://:9092
- KAFKA_CFG_ADVERTISED_LISTENERS=SASL_SSL://:9092
- KAFKA_ZOOKEEPER_USER=kafka
- KAFKA_ZOOKEEPER_PASSWORD=kafka_password
In order to get the required credentials to consume and produce messages you need to provide the credentials in the client. If your Kafka client allows it, use the credentials you've provided.
While producing and consuming messages using the bitnami/kafka image, you'll need to point to the
consumer.properties and/or producer.properties file, which contains the needed configuration
to work. You can find this files in the /opt/bitnami/kafka/conf directory
Use this to generate messages using a secure setup
export KAFKA_OPTS="-Djava.security.auth.login.config=/opt/bitnami/kafka/conf/kafka_jaas.conf"
kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic test --producer.config /opt/bitnami/kafka/conf/producer.properties
Use this to consume messages using a secure setup
export KAFKA_OPTS="-Djava.security.auth.login.config=/opt/bitnami/kafka/conf/kafka_jaas.conf"
kafka-console-consumer.sh --bootstrap-server 127.0.0.1:9092 --topic test --consumer.config /opt/bitnami/kafka/conf/consumer.properties
If you use other tools to use your Kafka cluster, you'll need to provide the required information.
You can find the required information in the files located at /opt/bitnami/kafka/conf directory.
A Kafka cluster can easily be setup with the Bitnami Kafka Docker image using the following environment variables:
KAFKA_CFG_ZOOKEEPER_CONNECT: Comma separated host:port pairs, each corresponding to a Zookeeper Server.Create a Docker network to enable visibility to each other via the docker container name
docker network create app-tier --driver bridge
The first step is to create one Zookeeper instance.
docker run --name zookeeper \
--network app-tier \
-e ALLOW_ANONYMOUS_LOGIN=yes \
-p 2181:2181 \
bitnami/zookeeper:latest
The first step is to create one Kafka instance.
docker run --name kafka1 \
--network app-tier \
-e KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181 \
-e ALLOW_PLAINTEXT_LISTENER=yes \
-p 9092:9092 \
bitnami/kafka:latest
Next we start a new Kafka container.
docker run --name kafka2 \
--network app-tier \
-e KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181 \
-e ALLOW_PLAINTEXT_LISTENER=yes \
-p 9092:9092 \
bitnami/kafka:latest
Next we start another new Kafka container.
docker run --name kafka3 \
--network app-tier \
-e KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181 \
-e ALLOW_PLAINTEXT_LISTENER=yes \
-p 9092:9092 \
bitnami/kafka:latest
You now have a Kafka cluster up and running. You can scale the cluster by adding/removing slaves without incurring any downtime.
With Docker Compose, topic replication can be setup using:
version: '2'
services:
zookeeper:
image: 'bitnami/zookeeper:latest'
ports:
- '2181:2181'
environment
- ALLOW_ANONYMOUS_LOGIN=yes
kafka1:
image: 'bitnami/kafka:latest'
ports:
- '9092'
environment:
- KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
- ALLOW_PLAINTEXT_LISTENER=yes
kafka2:
image: 'bitnami/kafka:latest'
ports:
- '9092'
environment:
- KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
- ALLOW_PLAINTEXT_LISTENER=yes
kafka3:
image: 'bitnami/kafka:latest'
ports:
- '9092'
environment:
- KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
- ALLOW_PLAINTEXT_LISTENER=yes
Then, you can create a replicated topic with:
root@kafka1:/# /opt/bitnami/kafka/bin/kafka-topics.sh --create --zookeeper zookeeper:2181 --topic mytopic --partitions 3 --replication-factor 3
Created topic "mytopic".
root@kafka1:/# /opt/bitnami/kafka/bin/kafka-topics.sh --describe --zookeeper zookeeper:2181 --topic mytopic
Topic:mytopic PartitionCount:3 ReplicationFactor:3 Configs:
Topic: mytopic Partition: 0 Leader: 2 Replicas: 2,3,1 Isr: 2,3,1
Topic: mytopic Partition: 1 Leader: 3 Replicas: 3,1,2 Isr: 3,1,2
Topic: mytopic Partition: 2 Leader: 1 Replicas: 1,2,3 Isr: 1,2,3
The image looks for configuration in the conf/ directory of /opt/bitnami/kafka.
docker run --name kafka -v /path/to/server.properties:/opt/bitnami/kafka/conf/server.properties bitnami/kafka:latest
After that, your changes will be taken into account in the server's behaviour.
Run the Kafka image, mounting a directory from your host.
Modify the docker-compose.yml file present in this repository:
kafka:
volumes:
- /path/to/server.properties:/opt/bitnami/kafka/conf/server.properties
Edit the configuration on your host using your favorite editor.
vi /path/to/server.properties
After changing the configuration, restart your Kafka container for changes to take effect.
docker restart kafka
Or using Docker Compose:
docker-compose restart kafka
The Bitnami Kafka Docker image sends the container logs to the stdout. To view the logs:
docker logs kafka
Or using Docker Compose:
docker-compose logs kafka
You can configure the containers logging driver using the --log-driver option if you wish to consume the container logs differently. In the default configuration docker uses the json-file driver.
To backup your data, configuration and logs, follow these simple steps:
docker stop kafka
Or using Docker Compose:
docker-compose stop kafka
We need to mount two volumes in a container we will use to create the backup: a directory on your host to store the backup in, and the volumes from the container we just stopped so we can access the data.
docker run --rm -v /path/to/kafka-backups:/backups --volumes-from kafka busybox \
cp -a /bitnami/kafka:latest /backups/latest
Or using Docker Compose:
docker run --rm -v /path/to/kafka-backups:/backups --volumes-from `docker-compose ps -q kafka` busybox \
cp -a /bitnami/kafka:latest /backups/latest
Restoring a backup is as simple as mounting the backup as volumes in the container.
docker run -v /path/to/kafka-backups/latest:/bitnami/kafka bitnami/kafka:latest
You can also modify the docker-compose.yml file present in this repository:
kafka:
volumes:
- /path/to/kafka-backups/latest:/bitnami/kafka
Bitnami provides up-to-date versions of Kafka, including security patches, soon after they are made upstream. We recommend that you follow these steps to upgrade your container.
docker pull bitnami/kafka:latest
or if you're using Docker Compose, update the value of the image property to
bitnami/kafka:latest.
Before continuing, you should backup your container's data, configuration and logs.
Follow the steps on creating a backup.
docker rm -v kafka
Or using Docker Compose:
docker-compose rm -v kafka
Re-create your container from the new image, restoring your backup if necessary.
docker run --name kafka bitnami/kafka:latest
Or using Docker Compose:
docker-compose up kafka
The following environment variables were beingly wrongly translated into KAFKA_CFG_ environment variables, and therefore they were being wrongly mapped into Kafka keys:
KAFKA_LOGS_DIRS -> KAFKA_CFG_LOG_DIRSKAFKA_PORT_NUMBER -> KAFKA_CFG_PORTKAFKA_ZOOKEEPER_CONNECT_TIMEOUT_MS -> KAFKA_CFG_ZOOKEEPER_CONNECTION_TIMEOUT_MSFor consistency reasons with previous environment variables, the following KAFKA_ to KAFKA_CFG_ environment variable translations are now supported for mapping into Kafka keys:
KAFKA_LOG_DIRS -> KAFKA_CFG_LOG_DIRSKAFKA_ZOOKEEPER_CONNECTION_TIMEOUT_MS -> KAFKA_CFG_ZOOKEEPER_CONNECTION_TIMEOUT_MSKAFKA_CFG_, as they are now mapped directly to Kafka keys. Variables changed:
KAFKA_ADVERTISED_LISTENERS -> KAFKA_CFG_ADVERTISED_LISTENERSKAFKA_BROKER_ID -> KAFKA_CFG_BROKER_IDKAFKA_DEFAULT_REPLICATION_FACTOR -> KAFKA_CFG_DEFAULT_REPLICATION_FACTORKAFKA_DELETE_TOPIC_ENABLE -> KAFKA_CFG_DELETE_TOPIC_ENABLEKAFKA_INTER_BROKER_LISTENER_NAME -> KAFKA_CFG_INTER_BROKER_LISTENER_NAMEKAFKA_LISTENERS -> KAFKA_CFG_LISTENERSKAFKA_LISTENER_SECURITY_PROTOCOL_MAP -> KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAPKAFKA_LOGS_DIRS -> KAFKA_CFG_LOG_DIRSKAFKA_LOG_FLUSH_INTERVAL_MESSAGES -> KAFKA_CFG_LOG_FLUSH_INTERVAL_MESSAGESKAFKA_LOG_FLUSH_INTERVAL_MS -> KAFKA_CFG_LOG_FLUSH_INTERVAL_MSKAFKA_LOG_MESSAGE_FORMAT_VERSION -> KAFKA_CFG_LOG_MESSAGE_FORMAT_VERSIONKAFKA_LOG_RETENTION_BYTES -> KAFKA_CFG_LOG_RETENTION_BYTESKAFKA_LOG_RETENTION_CHECK_INTERVALS_MS -> KAFKA_CFG_LOG_RETENTION_CHECK_INTERVAL_MSKAFKA_LOG_RETENTION_HOURS -> KAFKA_CFG_LOG_RETENTION_HOURSKAFKA_MAX_MESSAGE_BYTES -> KAFKA_CFG_MESSAGE_MAX_BYTESKAFKA_NUM_IO_THREADS -> KAFKA_CFG_NUM_IO_THREADSKAFKA_NUM_NETWORK_THREADS -> KAFKA_CFG_NUM_NETWORK_THREADSKAFKA_NUM_PARTITIONS -> KAFKA_CFG_NUM_PARTITIONSKAFKA_NUM_RECOVERY_THREADS_PER_DATA_DIR -> KAFKA_CFG_NUM_RECOVERY_THREADS_PER_DATA_DIRKAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR -> KAFKA_CFG_OFFSETS_TOPIC_REPLICATION_FACTORKAFKA_PORT -> KAFKA_CFG_PORTKAFKA_SEGMENT_BYTES -> KAFKA_CFG_SEGMENT_BYTESKAFKA_SOCKET_RECEIVE_BUFFER_BYTES -> KAFKA_CFG_SOCKET_RECEIVE_BUFFER_BYTESKAFKA_SOCKET_REQUEST_MAX_BYTES -> KAFKA_CFG_SOCKET_REQUEST_MAX_BYTESKAFKA_SOCKET_SEND_BUFFER_BYTES -> KAFKA_CFG_SOCKET_SEND_BUFFER_BYTESKAFKA_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM -> KAFKA_CFG_SSL_ENDPOINT_IDENTIFICATION_ALGORITHMKAFKA_TRANSACTION_STATE_LOG_MIN_ISR -> KAFKA_CFG_TRANSACTION_STATE_LOG_MIN_ISRKAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR -> KAFKA_CFG_TRANSACTION_STATE_LOG_REPLICATION_FACTORKAFKA_ZOOKEEPER_CONNECT_TIMEOUT_MS -> KAFKA_CFG_ZOOKEEPER_CONNECT_TIMEOUT_MSKAFKA_ZOOKEEPER_CONNECT -> KAFKA_CFG_ZOOKEEPER_CONNECTroot user and the kafka daemon was started as kafka user. From now own, both the container and the kafka daemon run as user 1001.
As a consequence, the configuration files are writable by the user running the kafka process.We'd love for you to contribute to this container. You can request new features by creating an issue, or submit a pull request with your contribution.
If you encountered a problem running this container, you can file an issue. For us to provide better support, be sure to include the following information in your issue:
docker version)docker infoecho $BITNAMI_IMAGE_VERSION inside the container)Copyright (c) 2015-2019 Bitnami
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
此处可能存在不合适展示的内容，页面不予展示。您可通过相关编辑功能自查并修改。
如您确认内容无涉及 不当用语 / 纯广告导流 / 暴力 / 低俗色情 / 侵权 / 盗版 / 虚假 / 无价值内容或违法国家有关法律法规的内容，可点击提交进行申诉，我们将尽快为您处理。
马建仓 AI 助手
