使用 docker-compose 和端口映射进行本地 Kafka 测试

Question

我正在尝试使用 来启动和运行黑盒服务测试的基础设施docker-compose。我想在 Jenkins 上为多个服务执行此操作，因此有必要绑定到特定于构建的端口（例如 50012 而不是 9092），以便构建应该能够并行运行。

问题是，生成消息失败。

这是我的docker-compose.yml：

---
version: "3.4"

services:

  zookeeper:
    image: "confluentinc/cp-zookeeper:latest"
    ports:
      - "2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    extra_hosts:
      - "moby:127.0.0.1"

  kafka:
    image: "confluentinc/cp-kafka:latest"
    environment:
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_PROTOCOL_NAME: INSIDE
      KAFKA_ADVERTISED_PROTOCOL_NAME: OUTSIDE
      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
      KAFKA_LISTENERS: "OUTSIDE://localhost:50012,INSIDE://kafka:9092"
      KAFKA_ADVERTISED_LISTENERS: "OUTSIDE://localhost:50012,INSIDE://kafka:9092"
      KAFKA_BROKER_ID: 1
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
    ports:
      - "50012:9092"
    extra_hosts:
      - "moby:127.0.0.1"
    depends_on:
      - zookeeper

除了这个错误（我认为可以忽略）之外，我没有看到任何错误：

ERROR Could not submit metrics to Kafka topic __confluent.support.metrics: Failed to construct kafka producer (io.confluent.support.metrics.BaseMetricsReporter)

当我尝试像这样用 Kafka 控制台生产者发送消息时

echo 'my-message' | kafka-console-producer.sh --broker-list localhost:50012 --topic test

我收到此异常并且消息未发送：

[2018-07-25 10:56:31,283] ERROR Error when sending message to topic test with key: null, value: 10 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)

org.apache.kafka.common.errors.TimeoutException：60000 毫秒后无法更新元数据。

---

更新：感谢 Robin 的正确答案，我们能够使用以下 docker-compose.yml 运行它：

---
version: "3.4"

services:

zookeeper:
    image: "confluentinc/cp-zookeeper:latest"
    environment:
    ZOOKEEPER_CLIENT_PORT: 2181
    ZOOKEEPER_TICK_TIME: 2000
    extra_hosts:
      - "moby:127.0.0.1"

kafka:
    image: "confluentinc/cp-kafka:latest"
    environment:
    KAFKA_BROKER_ID: 1
    KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
    KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: "INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT"
    KAFKA_INTER_BROKER_LISTENER_NAME: "INSIDE"
    KAFKA_ADVERTISED_LISTENERS: "INSIDE://kafka:29092,OUTSIDE://localhost:50012"
    KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
    KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
    ports:
      - "50012:50012"
    extra_hosts:
      - "moby:127.0.0.1"
    depends_on:
      - zookeeper

Answer 1

你的配置的问题是

\n\n

      KAFKA_ADVERTISED_LISTENERS: "OUTSIDE://localhost:50012,INSIDE://kafka:9092"\n[\xe2\x80\xa6]\n    ports:\n      - "50012:9092"\n

\n\n

这KAFKA_ADVERTISED_LISTENERS正是经纪人公布的地址。因此，一方面您将其配置为localhost:50012，但另一方面您从 Docker 配置中将其公开9092为外部端口。

\n\n

如果你只是使用

\n\n

 ports:\n      - "50012:50012"\n

\n\n

那么它应该可以正常工作。

\n\n

查看此docker-compose，了解在外部和内部公开 Kafka 的工作配置示例。

\n