Elasticsearch 未分配的分片 CircuitBreakingException[[parent] 数据太大

Question

我收到警报，指出 elasticsearch 有 2 个未分配的分片。我进行了以下 api 调用以收集更多详细信息。

    curl -s http://localhost:9200/_cluster/allocation/explain | python -m json.tool

下面的输出

    "allocate_explanation": "cannot allocate because allocation is not permitted to any of the nodes",
    "can_allocate": "no",
    "current_state": "unassigned",
    "index": "docs_0_1603929645264",
    "node_allocation_decisions": [
        {
            "deciders": [
                {
                    "decider": "max_retry",
                    "decision": "NO",
                    "explanation": "shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [/_cluster/reroute?retry_failed=true] to retry, [unassigned_info[[reason=ALLOCATION_FAILED], at[2020-10-30T06:10:16.305Z], failed_attempts[5], delayed=false, details[failed shard on node [o_9jyrmOSca9T12J4bY0Nw]: failed recovery, failure RecoveryFailedException[[docs_0_1603929645264][0]: Recovery failed from {elasticsearch-data-1}{fIaSuZsNTwODgZnt90f7kQ}{Qxl9iPacQVS-tN_t4YJqrw}{IP1}{IP:9300} into {elasticsearch-data-0}{o_9jyrmOSca9T12J4bY0Nw}{1w5mgwy0RYqBQ9c-qA_6Hw}{IP}{IP:9300}]; nested: RemoteTransportException[[elasticsearch-data-1][IP:9300][internal:index/shard/recovery/start_recovery]]; nested: RecoveryEngineException[Phase[1] phase1 failed]; nested: RecoverFilesRecoveryException[Failed to transfer [129] files with total size of [4.4gb]]; nested: RemoteTransportException[[elasticsearch-data-0][IP2:9300][internal:index/shard/recovery/file_chunk]]; nested: 
CircuitBreakingException[[parent] Data too large, data for [<transport_request>] would be [1972835086/1.8gb], which is larger than the limit of [1972122419/1.8gb], real usage: [1972833976/1.8gb], new bytes reserved: [1110/1kb]]; ], allocation_status[no_attempt]]]"
                }
            ],
            "node_decision": "no",
            "node_id": "1XEXS92jTK-asdfasdfasdf",
            "node_name": "elasticsearch-data-2",
            "transport_address": "IP1:9300"
        },
        {
            "deciders": [
                {
                    "decider": "max_retry",
                    "decision": "NO",
                    "explanation": "shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [/_cluster/reroute?retry_failed=true] to retry, [unassigned_info[[reason=ALLOCATION_FAILED], at[2020-10-30T06:10:16.305Z], failed_attempts[5], delayed=false, details[failed shard on node [o_9jyrmOSca9T12J4bY0Nw]: failed recovery, failure RecoveryFailedException[[docs_0_1603929645264][0]: Recovery failed from {elasticsearch-data-1}{fIaSuZsNTwODgZnt90f7kQ}{Qxl9iPacQVS-tN_t4YJqrw}{IP1}{IP1:9300} into {elasticsearch-data-0}{o_9jyrmOSca9T12J4bY0Nw}{1w5mgwy0RYqBQ9c-qA_6Hw}{IP2}{IP2:9300}]; nested: RemoteTransportException[[elasticsearch-data-1][IP1:9300][internal:index/shard/recovery/start_recovery]]; nested: RecoveryEngineException[Phase[1] phase1 failed]; nested: RecoverFilesRecoveryException[Failed to transfer [129] files with total size of [4.4gb]]; nested: RemoteTransportException[[elasticsearch-data-0][IP2:9300][internal:index/shard/recovery/file_chunk]]; nested: 
CircuitBreakingException[[parent] Data too large, data for [<transport_request>] would be [1972835086/1.8gb], which is larger than the limit of [1972122419/1.8gb], real usage: [1972833976/1.8gb], new bytes reserved: [1110/1kb]]; ], allocation_status[no_attempt]]]"
                },
                {
                    "decider": "same_shard",
                    "decision": "NO",
                    "explanation": "the shard cannot be allocated to the same node on which a copy of the shard already exists [[docs_0_1603929645264][0], node[fIaSuZsNTwODgZnt90f7kQ], [P], s[STARTED], a[id=stHnyqjLQ7OwFbaqs5vWqA]]"
                }
            ],
            "node_decision": "no",
            "node_id": "fIaSuZsNTwODgZnt90f7kQ",
            "node_name": "elasticsearch-data-1",
            "transport_address": "IP1:9300"
        },
        {
            "deciders": [
                {
                    "decider": "max_retry",
                    "decision": "NO",
                    "explanation": "shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [/_cluster/reroute?retry_failed=true] to retry, [unassigned_info[[reason=ALLOCATION_FAILED], at[2020-10-30T06:10:16.305Z], failed_attempts[5], delayed=false, details[failed shard on node [o_9jyrmOSca9T12J4bY0Nw]: failed recovery, failure RecoveryFailedException[[docs_0_1603929645264][0]: Recovery failed from {elasticsearch-data-1}{fIaSuZsNTwODgZnt90f7kQ}{Qxl9iPacQVS-tN_t4YJqrw}{IP1}{IP1:9300} into {elasticsearch-data-0}{o_9jyrmOSca9T12J4bY0Nw}{1w5mgwy0RYqBQ9c-qA_6Hw}{Ip2}{IP2:9300}]; nested: RemoteTransportException[[elasticsearch-data-1][IP1:9300][internal:index/shard/recovery/start_recovery]]; nested: RecoveryEngineException[Phase[1] phase1 failed]; nested: RecoverFilesRecoveryException[Failed to transfer [129] files with total size of [4.4gb]]; nested: RemoteTransportException[[elasticsearch-data-0][IP2:9300][internal:index/shard/recovery/file_chunk]]; nested: 
CircuitBreakingException[[parent] Data too large, data for [<transport_request>] would be [1972835086/1.8gb], which is larger than the limit of [1972122419/1.8gb], real usage: [1972833976/1.8gb], new bytes reserved: [1110/1kb]]; ], allocation_status[no_attempt]]]"
                }
            ],
            "node_decision": "no",
            "node_id": "o_9jyrmOSca9T12J4bY0Nw",
            "node_name": "elasticsearch-data-0",
            "transport_address": "IP2:9300"
        }
    ],
    "primary": false,
    "shard": 0,
    "unassigned_info": {
        "at": "2020-10-30T06:10:16.305Z",
        "details": "failed shard on node [o_9jyrmOSca9T12J4bY0Nw]: failed recovery, failure RecoveryFailedException[[docs_0_1603929645264][0]: Recovery failed from {elasticsearch-data-1}{fIaSuZsNTwODgZnt90f7kQ}{Qxl9iPacQVS-tN_t4YJqrw}{IP1}{IP1:9300} into {elasticsearch-data-0}{o_9jyrmOSca9T12J4bY0Nw}{1w5mgwy0RYqBQ9c-qA_6Hw}{IP2}{IP2:9300}]; nested: RemoteTransportException[[elasticsearch-data-1][IP1:9300][internal:index/shard/recovery/start_recovery]]; nested: RecoveryEngineException[Phase[1] phase1 failed]; nested: RecoverFilesRecoveryException[Failed to transfer [129] files with total size of [4.4gb]]; nested: RemoteTransportException[[elasticsearch-data-0][IP2:9300][internal:index/shard/recovery/file_chunk]]; nested: 
CircuitBreakingException[[parent] Data too large, data for [<transport_request>] would be [1972835086/1.8gb], which is larger than the limit of [1972122419/1.8gb], real usage: [1972833976/1.8gb], new bytes reserved: [1110/1kb]]; ",
        "failed_allocation_attempts": 5,
        "last_allocation_status": "no_attempt",
        "reason": "ALLOCATION_FAILED"
    }
}

我查询了断路器配置

    curl -X GET "localhost:9200/_nodes/stats/breaker?pretty

并且可以看到3个节点（elasticsearch-data-0、elasticsearch-data-1和elasticsearch-data-2）的父limit_size_in_byes如下。

"parent" : {
          "limit_size_in_bytes" : 1972122419,
          "limit_size" : "1.8gb",
          "estimated_size_in_bytes" : 1648057776,
          "estimated_size" : "1.5gb",
          "overhead" : 1.0,
          "tripped" : 139
        }

我参考了这个答案/sf/answers/4336808591/并计划增加断路器的内存百分比或整个 JVM 堆。

这是一个 k8s 环境，elasticsearch-data 部署为具有 3 个副本的 statefulset。当我描述 statefulset 时，我可以看到下面定义的 ENV 变量

Containers:
   elasticsearch:
    Image:      custom/elasticsearch-oss-s3:7.0.0
    Port:       9300/TCP
    Host Port:  0/TCP
    Limits:
      cpu:     10500m
      memory:  21Gi
    Requests:
      cpu:      10
      memory:   20Gi
    Environment:
      DISCOVERY_SERVICE:     elasticsearch-discovery
      NODE_MASTER:           false
      PROCESSORS:            11 (limits.cpu)
      ES_JAVA_OPTS:          -Djava.net.preferIPv4Stack=true -Xms2048m -Xmx2048m

按照这个，堆大小似乎是 2048m

我登录了一个 elasticsearch-data pod，在弹性搜索配置目录下我看到了下面的文件

elasticsearch.keystore  elasticsearch.yml  jvm.options  log4j2.properties  repository-s3

elasticsearch.yml 没有任何堆配置等。它只有主节点的名称等。

下面是 jvm.options 文件

## JVM configuration

# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space

-Xms1g
-Xmx1g


## GC configuration
-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly


## DNS cache policy
-Des.networkaddress.cache.ttl=60
-Des.networkaddress.cache.negative.ttl=10


# pre-touch memory pages used by the JVM during initialization
-XX:+AlwaysPreTouch

## basic

# explicitly set the stack size
-Xss1m

# set to headless, just in case
-Djava.awt.headless=true

# ensure UTF-8 encoding by default (e.g. filenames)
-Dfile.encoding=UTF-8

# use our provided JNA always versus the system one
-Djna.nosys=true

# turn off a JDK optimization that throws away stack traces for common
# exceptions because stack traces are important for debugging
-XX:-OmitStackTraceInFastThrow


# flags to configure Netty
-Dio.netty.noUnsafe=true
-Dio.netty.noKeySetOptimization=true
-Dio.netty.recycler.maxCapacityPerThread=0

# log4j 2
-Dlog4j.shutdownHookEnabled=false
-Dlog4j2.disable.jmx=true

-Djava.io.tmpdir=${ES_TMPDIR}

## heap dumps

# generate a heap dump when an allocation from the Java heap fails
# heap dumps are created in the working directory of the JVM
-XX:+HeapDumpOnOutOfMemoryError

# specify an alternative path for heap dumps; ensure the directory exists and
# has sufficient space
-XX:HeapDumpPath=data

# specify an alternative path for JVM fatal error logs
-XX:ErrorFile=logs/hs_err_pid%p.log

## JDK 8 GC logging

8:-XX:+PrintGCDetails
8:-XX:+PrintGCDateStamps
8:-XX:+PrintTenuringDistribution
8:-XX:+PrintGCApplicationStoppedTime
8:-Xloggc:logs/gc.log
8:-XX:+UseGCLogFileRotation
8:-XX:NumberOfGCLogFiles=32
8:-XX:GCLogFileSize=64m

# JDK 9+ GC logging
9-:-Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m
# due to internationalization enhancements in JDK 9 Elasticsearch need to set the provider to COMPAT otherwise
# time/date parsing will break in an incompatible way for some date patterns and locals
9-:-Djava.locale.providers=COMPAT

从上面看，总堆大小似乎是 1g。

但是从这个pod的stateful set中定义的env变量来看，好像是2048m。

哪一个是对的？

现在，从下面的链接

断路器设置 | 弹性搜索

可以使用以下设置配置父级断路器：

index.breaker.total.use_real_memory（静态）确定父断路器是应该考虑实际内存使用情况（真）还是只考虑子断路器保留的数量（假）。默认为真。

index.breaker.total.limit（动态）整个父断路器的启动限制。如果index.breaker.total.use_real_memory 为false，则默认为JVM 堆的70%。如果 index.breaker.total.use_real_memory 为 true，则默认为 JVM 堆的 95%。

但是我查询的错误和断路器统计信息中的限制值是 - 1972122419 字节（1.8G）。这似乎不是 2048M 或 1g 的 95%。

现在，如何增加断路器父的堆或内存限制，以便摆脱此错误？

Answer 1

这里有两件事，分片分配异常和断路器异常（看起来是嵌套异常）。

请在您的集群中使用以下命令重新触发分配，因为之前所有重试都失败了，如果您仔细注意到，在您的异常消息中也建议这样做。有关以下命令的更多信息在此相关的 Github 问题评论中。

curl -XPOST ':9200/_cluster/reroute?retry_failed

如果还是不行，那么你必须修复父断路器异常，你应该使用http://localhost:9200/_nodes/statsAPI来知道你的ES节点的确切堆，并相应地增加它。