Docker 中的 Kafka 服务器问题

Question

我正在 Windows 机器中使用 docker 来运行我的示例 Spark + Kafka 项目。\n我面临着

\n

WARN ClientUtils: Couldn't resolve server kafka:9092 from bootstrap.servers as DNS resolution failed for kafka\n[error] (run-main-0) org.apache.kafka.common.KafkaException: Failed to construct kafka consumer\n[error] org.apache.kafka.common.KafkaException: Failed to construct kafka consumer\n------\nCaused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers\n

\n

下面是我的 docker-compose.yml

\n

version: '2'\nservices:\n    test1:\n        build: test1service/.\n        depends_on:\n            - kafka\n    test2:\n        build: test2/.\n        depends_on:\n            - kafka\n            - test1\n    zookeeper:\n        image: confluentinc/cp-zookeeper:5.1.0\n        ports:\n         - 2181:2181\n        environment:\n          ZOOKEEPER_CLIENT_PORT: 2181\n          ZOOKEEPER_TICK_TIME: 2000\n        extra_hosts:\n          - "localhost: 127.0.0.1"\n    kafka:\n        image: confluentinc/cp-kafka:5.1.0\n        ports:\n         - 9092:9092\n        depends_on:\n          - zookeeper\n        environment:\n          KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181\n          KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092\n          #KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka:9092\n          KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"\n          KAFKA_DELETE_TOPIC_ENABLE: "true"\n          KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1\n        extra_hosts:\n          - "localhost: 127.0.0.1"\n

\n

以下是我在 test2 服务中的示例代码

\n

val inputStreamDF = spark.readStream.format("kafka").option("kafka.bootstrap.servers", "kafka:9092")\n                           .option("subscribe", "test1")\n                           .option("startingOffsets", "earliest")\n                           .load()\n

\n

docker ps 命令输出是

\n

CONTAINER ID   IMAGE                             COMMAND                  CREATED             STATUS          PORTS                                        NAMES\n4c8feb49e12b   test1                             "/usr/bin/start.sh"      About an hour ago   Up 50 minutes                                                test1\n4535ce246541   test2                             "/usr/bin/myservice-\xe2\x80\xa6"   About an hour ago   Up 50 minutes                                                test2\n733766f72adb   confluentinc/cp-kafka:5.1.0       "/etc/confluent/dock\xe2\x80\xa6"   About an hour ago   Up 51 minutes   0.0.0.0:9092->9092/tcp       kafka_1\nd915e25cb226   confluentinc/cp-zookeeper:5.1.0   "/etc/confluent/dock\xe2\x80\xa6"   About an hour ago   Up 51 minutes   2888/tcp, 0.0.0.0:2181->2181/tcp, 3888/tcp   zookeeper_1\n

\n

有没有人遇到过类似的问题，你是如何解决的？

\n

Answer 1

您尝试访问 kafka:9092，但 docker compose 已生成容器 kafka_1，这就是没有名称解析的原因。

Docker 为您的容器提供内部 IP，并使用其容器名称在此网络上创建内部 DNS（使用嵌入的 dns 服务器）

Docker Compose 不会更改您的环境变量以适应它为容器提供的名称。

您应该在容器描述中使用“container_name: kafka”来获取静态容器名称。

Answer 2

您唯一需要的是网络。之后所有服务都将连接，您可以使用kafka:9092。

version: '2'
services:
    test1:
        build: test1service/.
        depends_on:
            - kafka
        networks:
          - app-network
    test2:
        build: test2/.
        depends_on:
            - kafka
            - test1
        networks:
          - app-network
    zookeeper:
        image: confluentinc/cp-zookeeper:5.1.0
        ports:
         - 2181:2181
        environment:
          ZOOKEEPER_CLIENT_PORT: 2181
          ZOOKEEPER_TICK_TIME: 2000
        extra_hosts:
          - "localhost: 127.0.0.1"
        networks:
          - app-network
    kafka:
        image: confluentinc/cp-kafka:5.1.0
        ports:
         - 9092:9092
        depends_on:
          - zookeeper
        environment:
          KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
          KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
          #KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka:9092
          KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
          KAFKA_DELETE_TOPIC_ENABLE: "true"
          KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
        extra_hosts:
          - "localhost: 127.0.0.1"
        networks:
          - app-network
networks:
  app-network:
    driver: bridge