Docker compose 创建 kafka 主题

Question

问题：无法从 docker-compose 创建主题。在运行测试系统之前，我需要创建 kafka 主题。计划将其用作管道的一部分，因此使用 UI 不是一个选项。

\n

注意：kafka 需要大约 15 秒才能准备好，因此我需要在添加主题之前休眠 15 秒。

\n

可能的解决方案：

\n

\n
1. 创建一个 shell.sh 文件，其中包含等待 15 秒的命令，然后添加一堆主题
\n
2. 为其创建一个 dockerfile
\n
3. 在启动测试系统之前将该 docker 镜像包含在 docker-compose.yml 中
\n

\n

电流：

\n

\n
1. 创建动物园管理员 - 确定
\n
2. 创建 kafka1 - 确定
\n
3. 休息代理-好的
\n
4. 创建主题<- 问题
\n
5. 创建 SUT - 确定
\n

\n

当前 docker-compose.yml：

\n

version: '2'\nservices:\nzookeeper:\nimage: docker.io/confluentinc/cp-zookeeper:5.4.1\nhostname: zookeeper\ncontainer_name: zookeeper\nports:\n- "2181:2181"\nenvironment:\nZOOKEEPER_CLIENT_PORT: 2181\nZOOKEEPER_TICK_TIME: 2000\n\nKafka1:\nimage: docker.io/confluentinc/cp-enterprise-kafka:5.4.1\nhostname: Kafka1\ncontainer_name: Kafka1\ndepends_on:\n- zookeeper\nports:\n- "9092:9092"\n- "9101:9101"\nenvironment:\nKAFKA_BROKER_ID: 1\nKAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'\nKAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT\nKAFKA_ADVERTISED_HOST_NAME: Kafka1\nKAFKA_ADVERTISED_LISTENERS: PLAINTEXT://Kafka1:29092,PLAINTEXT_HOST://localhost:9092\nKAFKA_METRIC_REPORTERS: io.confluent.metrics.reporter.ConfluentMetricsReporter\nKAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1\nKAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0\nKAFKA_CONFLUENT_LICENSE_TOPIC_REPLICATION_FACTOR: 1\nKAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1\nKAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1\nKAFKA_JMX_PORT: 9101\nKAFKA_CONFLUENT_SCHEMA_REGISTRY_URL: http://schema-registry:8081\nCONFLUENT_METRICS_REPORTER_BOOTSTRAP_SERVERS: Kafka1:29092\nCONFLUENT_METRICS_REPORTER_ZOOKEEPER_CONNECT: zookeeper:2181\nCONFLUENT_METRICS_REPORTER_TOPIC_REPLICAS: 1\nCONFLUENT_METRICS_ENABLE: 'true'\nCONFLUENT_SUPPORT_CUSTOMER_ID: 'anonymous'\n\nrest-proxy:\nimage: docker.io/confluentinc/cp-kafka-rest:5.4.1\ndepends_on:\n- zookeeper\n- Kafka1\nports:\n- 8082:8082\nhostname: rest-proxy\ncontainer_name: rest-proxy\nenvironment:\nKAFKA_REST_HOST_NAME: rest-proxy\nKAFKA_REST_BOOTSTRAP_SERVERS: 'Kafka1:29092'\nKAFKA_REST_LISTENERS: "http://0.0.0.0:8082"\nKAFKA_REST_SCHEMA_REGISTRY_URL: 'http://schema-registry:8081'\n\n\ntopics:\nimage: topics:latest\nhostname: topics\ncontainer_name: topics\ndepends_on:\n- zookeeper\n- Kafka1\n- rest-proxy\n\n\nsut:\nimage: sut:latest\nhostname: sut\ncontainer_name: sut\ndepends_on:\n- zookeeper\n- Kafka1\n- rest-proxy\nports:\n- 5000:80\n

\n

主题容器的当前 Dockerfile：

\n

FROM ubuntu:14.04\n\nADD topics.sh /usr/local/bin/topics.sh\n\nRUN chmod +x /usr/local/bin/topics.sh\n\nCMD /usr/local/bin/topics.sh\n

\n

当前的topics.sh文件：

\n

#!/bin/sh\necho "Start: Sleep 15 seconds"\nsleep 30;\nwait;\necho "Begin creating topics"\ndocker exec Kafka1 kafka-topics --create --if-not-exists --zookeeper zookeeper:2181 --partitions 1 --replication-factor 1 --topic MY_AWESOME_TOPIC_ONE\ndocker exec Kafka1 kafka-topics --create --if-not-exists --zookeeper zookeeper:2181 --partitions 1 --replication-factor 1 --topic MY_AWESOME_TOPIC_TWO\necho "Done creating topics"\n

\n

电流输出：

\n

/usr/local/bin/topics.sh: 1: /usr/local/bin/topics.sh: \xc3\xaf\xc2\xbb\xc2\xbf#!/bin/sh: not found\nStart: Sleep 15 seconds\nBegin creating topics\n/usr/local/bin/topics.sh: 8: /usr/local/bin/topics.sh: docker: not found\n/usr/local/bin/topics.sh: 9: /usr/local/bin/topics.sh: docker: not found\n/usr/local/bin/topics.sh: 10: /usr/local/bin/topics.sh: docker: not found\n/usr/local/bin/topics.sh: 11: /usr/local/bin/topics.sh: docker: not found\n/usr/local/bin/topics.sh: 12: /usr/local/bin/topics.sh: docker: not found\n/usr/local/bin/topics.sh: 13: /usr/local/bin/topics.sh: docker: not found\n/usr/local/bin/topics.sh: 14: /usr/local/bin/topics.sh: docker: not found\n/usr/local/bin/topics.sh: 15: /usr/local/bin/topics.sh: docker: not found\nDone creating topics\n

\n

未创建主题。我被困住了。请帮忙。

\n

Answer 1

最简单的方法是在 docker-compose 文件（init-kafka在下面的示例中调用）内启动一个单独的容器来启动各种kafka-topics --create ...命令，同时首先通过简单地运行 使其等待 Kafka 可达kafka-topics --list ...。

像这样：

version: '2.1'
services:

  zookeeper:
    image: confluentinc/cp-zookeeper:6.1.1
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181

  # reachable on 9092 from the host and on 29092 from inside docker compose
  kafka:
    image: confluentinc/cp-kafka:6.1.1
    depends_on:
      - zookeeper
    ports:
      - '9092:9092'
    expose:
      - '29092'
    environment:
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: '1'
      KAFKA_MIN_INSYNC_REPLICAS: '1'

  init-kafka:
    image: confluentinc/cp-kafka:6.1.1
    depends_on:
      - kafka
    entrypoint: [ '/bin/sh', '-c' ]
    command: |
      "
      # blocks until kafka is reachable
      kafka-topics --bootstrap-server kafka:29092 --list

      echo -e 'Creating kafka topics'
      kafka-topics --bootstrap-server kafka:29092 --create --if-not-exists --topic my-topic-1 --replication-factor 1 --partitions 1
      kafka-topics --bootstrap-server kafka:29092 --create --if-not-exists --topic my-topic-2 --replication-factor 1 --partitions 1

      echo -e 'Successfully created the following topics:'
      kafka-topics --bootstrap-server kafka:29092 --list
      "

运行它时，init-kafka容器应该记录如下内容：

docker logs docker_init-kafka_1


[2021-10-12 02:00:28,728] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka/172.24.0.3:29092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2021-10-12 02:00:28,832] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka/172.24.0.3:29092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2021-10-12 02:00:29,033] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka/172.24.0.3:29092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2021-10-12 02:00:29,335] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka/172.24.0.3:29092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)


Creating kafka topics
Created topic my-topic-1.
Created topic my-topic-2.
Successfully created the following topics:
my-topic-1
my-topic-2

Answer 2

该解决方案允许从 docker-compse.yml 创建主题

• 参考各自kafka镜像服务的DockerFile

• 记下相应 dockerFile 镜像中的最后一个命令（镜像/镜像层的 DockerHub 存储库）

• 就我而言，对于图像blockconfluenceinc/cp-kafka:latest，启动 kafka 服务的最后一个命令是“/etc/confluence/docker/run”

• 因此，在您的 docker-compose.yml 中包含以下命令

  command: sh -c "((sleep 15 && kafka-topics --create --zookeeper zookeeper:2181 --replication-factor 1 --partitions 3 --topic topicName)&) && /etc/confluent/docker/run "

这将启动kafka服务，延迟15秒然后创建一个主题。

请注意，我们假设 kafka 服务完全运行需要 15 秒

    kafka:
        image: confluentinc/cp-kafka:latest
        depends_on:
            - zookeeper
        ports:
            - "29092:29092"
        environment:
            KAFKA_BROKER_ID: 1
            KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
            KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
            KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
            KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
            KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
        command: sh -c "((sleep 15 && kafka-topics --create --zookeeper zookeeper:2181 --replication-factor 1 --partitions 3 --topic quick-starter)&) && /etc/confluent/docker/run ">

Answer 3

ubuntu容器没有安装docker。

它也没有该kafka-topics命令，因此您应该重新使用cp-enterprise-kafka已经拉取的图像并将 ENTRYPOINT 或 CMD 更改为您的脚本，但直接运行 kafka-topics 命令

或者用 wurstmeister/kafka 替换 Kafka 容器，并添加用于创建主题的环境变量