Docker compose 不会批量保存数据

Question

我正在尝试使用 docker-compose 运行 Kafka。我得到了这个 yml 文件：

version: '3'


services:
  zookeeper:
    image: ${REPOSITORY}/cp-zookeeper:${TAG}
    hostname: zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    volumes:
      - ./zoo:/var/lib/zookeeper

  broker:
    image: ${REPOSITORY}/cp-kafka:${TAG}
    hostname: broker
    container_name: broker
    depends_on:
      - zookeeper
    ports:
      - "29092:29092"
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
    volumes:
      - ./broker:/var/lib/kafka

我在带有 docker-compose.yml 文件的目录中运行了一个命令：

docker-compose up -d

在该文件夹之后./broker并./zoo出现在我的目录中。在内部，它们具有类似于容器内部的结构 ( ./zoo/data, ./broker/data)。但是目录中没有文件。

我试过

docker-compose exec broker ls /var/lib/kafka/data

我看到了关于默认主题的文件夹和文件

Answer 1

这归结为卷之间的交互（如Dockerfile 中声明的那样），以及作为Docker Compose一部分的您要堆起来的卷。

如果您检查每个容器的 Dockerfile，您将看到它声明了卷，您也可以通过检查它来看到。这是使用您的配置时的样子：

? docker inspect zookeeper|jq '.[].Mounts[] | .Type ,.Destination'
"volume"
"/etc/zookeeper/secrets"
"bind"
"/var/lib/zookeeper"
"volume"
"/var/lib/zookeeper/log"
"volume"
"/var/lib/zookeeper/data"

您会注意到有两个卷（在映像本身中声明，即来自 Dockerfile）针对 ZK 的特定数据路径

• /var/lib/zookeeper/log
• /var/lib/zookeeper/data

此外，还有来自 Docker Compose 的绑定安装：

• /var/lib/zookeeper/

这些冲突，这就解释了你所看到的问题。

经纪人也存在类似的模式。

所以简而言之，您需要为映像中的每个特定卷挂载一个本地主机目录：

? docker inspect zookeeper|jq '.[].Mounts[] | .Type ,.Destination'
"volume"
"/etc/zookeeper/secrets"
"bind"
"/var/lib/zookeeper"
"volume"
"/var/lib/zookeeper/log"
"volume"
"/var/lib/zookeeper/data"

完成此操作后，我们可以看到容器路径中没有冲突：

? docker inspect zookeeper|jq '.[].Mounts '
[
  {
    "Type": "bind",
    "Source": "/private/tmp/zoo/log",
    "Destination": "/var/lib/zookeeper/log",
    "Mode": "rw",
    "RW": true,
    "Propagation": "rprivate"
  },
  {
    "Type": "bind",
    "Source": "/private/tmp/zoo/data",
    "Destination": "/var/lib/zookeeper/data",
    "Mode": "rw",
    "RW": true,
    "Propagation": "rprivate"
  },
  {
    "Type": "volume",
    "Name": "6cbb584e0d9aa2f119869b264544f587909d9f417fc553a7bb2954dd28ecb8ea",
    "Source": "/var/lib/docker/volumes/6cbb584e0d9aa2f119869b264544f587909d9f417fc553a7bb2954dd28ecb8ea/_data",
    "Destination": "/etc/zookeeper/secrets",
    "Driver": "local",
    "Mode": "",
    "RW": true,
    "Propagation": ""
  }
]

和来自容器的数据：

? docker exec zookeeper ls -l /var/lib/zookeeper/data /var/lib/zookeeper/log
/var/lib/zookeeper/data:
total 0
drwxr-xr-x 3 root root 96 Apr  3 08:59 version-2

/var/lib/zookeeper/log:
total 0
drwxr-xr-x 3 root root 96 Apr  3 08:59 version-2

? docker exec broker ls -l /var/lib/kafka/data
total 16
drwxr-xr-x 6 root root 192 Apr  3 08:59 __confluent.support.metrics-0
-rw-r--r-- 1 root root   0 Apr  3 08:59 cleaner-offset-checkpoint
-rw-r--r-- 1 root root   4 Apr  3 09:01 log-start-offset-checkpoint
-rw-r--r-- 1 root root  88 Apr  3 08:59 meta.properties
-rw-r--r-- 1 root root  36 Apr  3 09:01 recovery-point-offset-checkpoint
-rw-r--r-- 1 root root  36 Apr  3 09:02 replication-offset-checkpoint
-rw-r--r-- 1 root root   0 Apr  3 08:30 wibble

存储在本地主机上：

? ls -l broker/data zoo/data zoo/log
broker/data:
total 32
drwxr-xr-x  6 rmoff  wheel  192  3 Apr 09:59 __confluent.support.metrics-0
-rw-r--r--  1 rmoff  wheel    0  3 Apr 09:59 cleaner-offset-checkpoint
-rw-r--r--  1 rmoff  wheel    4  3 Apr 10:00 log-start-offset-checkpoint
-rw-r--r--  1 rmoff  wheel   88  3 Apr 09:59 meta.properties
-rw-r--r--  1 rmoff  wheel   36  3 Apr 10:00 recovery-point-offset-checkpoint
-rw-r--r--  1 rmoff  wheel   36  3 Apr 10:01 replication-offset-checkpoint
-rw-r--r--  1 rmoff  wheel    0  3 Apr 09:30 wibble

zoo/data:
total 0
drwxr-xr-x  3 rmoff  wheel  96  3 Apr 09:59 version-2

zoo/log:
total 0
drwxr-xr-x  3 rmoff  wheel  96  3 Apr 09:59 version-2

另请参阅Kafka 和 ZooKeeper 的数据卷