无需 Zookeeper 即可运行的 Kafka docker 镜像

Question

我读到Kafka不再需要zookeeper，所以我不想在docker-compose中使用zookeeper。但我不知道哪个卡夫卡图像可以在没有动物园管理员的情况下工作。谁能给出提示吗？

Answer 1

这是一个不需要 Zookeeper 的 Kafka Docker 镜像（如上所述）：

https://hub.docker.com/r/bashj79/kafka-kraft

免责声明：我是作者。

Answer 2

根据“What\xe2\x80\x99s Apache Kafka 3.3中的新功能”文档和“ KIP-833：将KRaft标记为生产就绪”，Kafka可以在没有Zookeeper的情况下工作（但有一些功能仅适用于Apache ZooKeeper（ZK）模式）。

\n

示例（docker-compose.yml）：

\n

version: "2.5"\nvolumes:\n  volume1:\nservices:\n  kafka1:\n    image: \'bitnami/kafka:3.3.1\'\n    container_name: kafka\n    environment:\n      - KAFKA_ENABLE_KRAFT=yes\n      - KAFKA_CFG_PROCESS_ROLES=broker,controller\n      - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER\n      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093\n      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT\n      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka1:9092\n      - KAFKA_CFG_BROKER_ID=1\n      - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=1@kafka1:9093\n      - ALLOW_PLAINTEXT_LISTENER=yes\n      - KAFKA_KRAFT_CLUSTER_ID=r4zt_wrqTRuT7W2NJsB_GA\n    volumes:\n      - volume1:/bitnami/kafka\n  kafka-ui:\n    container_name: kafka-ui\n    image: \'provectuslabs/kafka-ui:latest\'\n    ports:\n      - "8080:8080"\n    environment:\n      - KAFKA_CLUSTERS_0_BOOTSTRAP_SERVERS=kafka1:9092\n      - KAFKA_CLUSTERS_0_NAME=r4zt_wrqTRuT7W2NJsB_GA\n

\n

你可以尝试 localhost:8080 ，你会发现它工作得很好。

\n

Answer 3

Confluence在其存储库cp-all-in-one中发布了一个没有 Zookeeper 的工作docker-compose.yaml。

有一个脚本（update_run.sh）用作解决方法

#!/bin/sh

# Docker workaround: Remove check for KAFKA_ZOOKEEPER_CONNECT parameter
sed -i '/KAFKA_ZOOKEEPER_CONNECT/d' /etc/confluent/docker/configure

# Docker workaround: Ignore cub zk-ready
sed -i 's/cub zk-ready/echo ignore zk-ready/' /etc/confluent/docker/ensure

# KRaft required step: Format the storage directory with a new cluster ID
echo "kafka-storage format --ignore-formatted -t $(kafka-storage random-uuid) -c /etc/kafka/kafka.properties" >> /etc/confluent/docker/ensure

在docker-compose-setup命令中调用

  broker:
    image: confluentinc/cp-kafka:7.2.x-latest
    hostname: broker
    container_name: broker
    ports:
      - "9092:9092"
      - "9101:9101"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT'
      KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092'
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_JMX_PORT: 9101
      KAFKA_JMX_HOSTNAME: localhost
      KAFKA_PROCESS_ROLES: 'broker,controller'
      KAFKA_NODE_ID: 1
      KAFKA_CONTROLLER_QUORUM_VOTERS: '1@broker:29093'
      KAFKA_LISTENERS: 'PLAINTEXT://broker:29092,CONTROLLER://broker:29093,PLAINTEXT_HOST://0.0.0.0:9092'
      KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT'
      KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
      KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'
    volumes:
      - ./update_run.sh:/tmp/update_run.sh
    command: "bash -c 'if [ ! -f /tmp/update_run.sh ]; then echo \"ERROR: Did you forget the update_run.sh file that came with this docker-compose.yml file?\" && exit 1 ; else /tmp/update_run.sh && /etc/confluent/docker/run ; fi'"

Answer 4

您可以在没有 Zookeeper 的情况下使用此映像。

https://hub.docker.com/r/bitnami/kafka

这是一个 yaml 示例。

version: "3"
services:
  kafka:
    image: 'bitnami/kafka:3.2.3'
    restart: "no"
    privileged: true
    ports:
      - 2181:2181
      - 19092:19092
    environment:
      - KAFKA_ENABLE_KRAFT=yes
      - KAFKA_CFG_PROCESS_ROLES=broker,controller
      - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:19092,CONTROLLER://:2181
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://127.0.0.1:19092
      - KAFKA_BROKER_ID=1
      - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=1@127.0.0.1:2181
      - ALLOW_PLAINTEXT_LISTENER=yes

Answer 5

我读到卡夫卡不再需要动物园管理员

您可能已经读到，将来Apache Kafka 将不再需要 Zookeeper - 这在KIP-500中有详细介绍

然而，这尚未实现，因此暂时（2021 年 1 月）您的 Docker Compose 整体中仍然需要一个 Zookeeper。

Answer 6

到目前为止，最简单的方法是使用confluence-local图像。它使用此处定义的默认环境变量。

---
version: '2'
services:
  broker:
    image: confluentinc/confluent-local:7.4.1
    ports:
      - "8082:8082"
      - "9092:9092"
      - "9101:9101"

或者，如果您需要使用kafka-ui，您还应该配置一些环境变量（用于侦听器等）：

---
version: '2'
services:

  broker:
    image: confluentinc/confluent-local:7.4.1
    hostname: broker
    container_name: broker
    ports:
      - "8082:8082"
      - "9092:9092"
      - "9101:9101"
    environment:
      KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092'
      KAFKA_CONTROLLER_QUORUM_VOTERS: '1@broker:29093'
      KAFKA_LISTENERS: 'PLAINTEXT://broker:29092,CONTROLLER://broker:29093,PLAINTEXT_HOST://0.0.0.0:9092'

  kafka-ui:
    image: provectuslabs/kafka-ui:latest
    ports:
      - 9999:8080
    environment:
      DYNAMIC_CONFIG_ENABLED: true
      KAFKA_CLUSTERS_0_NAME: local
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: broker:29092