Elasticsearch Docker：超出洪水阶段磁盘水印 [95%]

Question

我的 Elasticsearch 节点在 docker 环境中运行时遇到问题。我用 docker-compose 启动它们，几分钟后它们告诉我： 洪水阶段磁盘水印 [95%] 超出

我在具有相当高存储容量的集群上运行它，并且我已经尝试增加 elasticsearch.yml 文件中的水印设置，但我仍然收到错误消息。也许这与 docker 容器的大小有关。

有谁知道可能是什么问题？任何帮助深表感谢。

docker-compose.yml 供参考：

version: '3.4'
services:
  es01:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.8.1
    container_name: es01
    environment:
      #- discovery.type=single-node
      - node.name=es01
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es02,es03
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - xpack.security.enabled=false
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data01:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    networks:
      - elastic

  es02:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.8.1
    container_name: es02
    environment:
      - node.name=es02
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es01,es03
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data02:/usr/share/elasticsearch/data
    networks:
      - elastic

  es03:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.8.1
    container_name: es03
    environment:
      - node.name=es03
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es01,es02
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data03:/usr/share/elasticsearch/data
    networks:
      - elastic

  kib01:
    image: docker.elastic.co/kibana/kibana:7.8.1
    container_name: kib01
    depends_on:
      - es01
      - es02
      - es03
    ports:
      - 5601:5601
    environment:
      ELASTICSEARCH_URL: http://es01:9200
      ELASTICSEARCH_HOSTS: http://es01:9200
    networks:
      - elastic

  client:
    image: appropriate/curl:latest
    depends_on:
      - es01
      - es02
      - es03
    networks:
      - elastic
    command: sh -c "curl es01:9200 && curl kib01:5601"

  dash_app:
    build: .
    ports:
    - 0.0.0.0:8050:8050
    depends_on:
      - es01
      - es02
      - es03
      - kib01
    networks:
      - elastic

#mapping:
#  image: appropriate/curl:latest
#  depends_on:
#    - es01
#    - es02
#    - es03
#  networks:
#    - elastic
#  command:    "curl -v -XPUT 'es01:9200/urteile' -H 'Content-Type: application/json' -d '
#       {
#         'mappings': {
#           'properties': {
#             'date': {
#               'type': 'date'
#             }
#           }
#         }
#       }
#    '"

  #web:
   # build: .
   # ports:
    #  - 8000:8000
    #depends_on:
    #  - es01
    #  - es02
    #  - es03
    #networks:
    #  - elastic


volumes:
  data01:
    driver: local
  data02:
    driver: local
  data03:
    driver: local

networks:
  elastic:
    driver: bridge

和码头信息：

Server:
 Containers: 6
  Running: 3
  Paused: 0
  Stopped: 3
 Images: 185
 Server Version: 19.03.12
 Storage Driver: overlay
  Backing Filesystem: extfs
  Supports d_type: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc nvidia
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.7.2-kd-cluster
 Operating System: Debian GNU/Linux 9 (stretch)
 OSType: linux
 Architecture: x86_64
 CPUs: 32
 Total Memory: 125.8GiB
 Name: dpl01
 ID: KBGO:2E6L:NIHR:UQAL:K5CN:XWBI:R7TK:WWZF:MZBT:BCHE:HUQW:UKKM
 Docker Root Dir: /data/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Answer 1

我找到了解决方案。问题与磁盘使用总量有关，如 sastorsl 的回答中所述： 低磁盘水印 [??%] 超过

我正在研究一个集群，其存储使用率为 98%，仍有 400GB 可用空间，但 Elasticsearch 只查看百分比，因此关闭了索引的任何写入权限。

解决方案是在节点启动后手动设置水印（在 elasticsearch.yml 中设置它们由于某种原因不起作用）：

curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_cluster/settings -d '{ "transient": { "cluster.routing.allocation.disk.threshold_enabled": false } }'
curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'

当然，您必须输入索引名称。之后，它们将再次可写。

Answer 2

就我而言，我是通过 docker-compose 中的 elasticsearch 8.7 和 ELK 得到的。

添加cluster.routing.allocation.disk.threshold_enabled=false到 ES 的服务docker-compose.yml解决了我的问题

    environment:
      - network.host=0.0.0.0
      - http.port=9200
      - transport.host=localhost
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - xpack.security.enabled=false
      - cluster.routing.allocation.disk.threshold_enabled=false   # <---

@Stimmot 的解决方案也对我有用，

curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_cluster/settings -d '{ "transient": { "cluster.routing.allocation.disk.threshold_enabled": false } }'

在健康检查中运行卷曲对我来说不起作用

  elasticsearch:

    healthcheck:
      test: ["CMD", "curl", "-XPUT", "-H", "'Content-Type: application/json'", "http://localhost:9200/_cluster/settings", "-d", "'{ \"transient\": { \"cluster.routing.allocation.disk.threshold_enabled\": false } }'"]
      interval: 10s
      timeout: 10s
      retries: 1 

  kivana:

    depends_on:
      elasticsearch:
        condition: service_healthy