PHP的Elasticsearch批量上传错误 - 已超出索引中的总字段数[1000]

Question

我们计划在我们的一个项目中使用ElasticSearch.目前,我们正在使用我们的数据测试ElasticSearch 5.0.1.我们面临的一个问题是,当我们从MySQL表格到弹性搜索的批量上传时,我们正在获取错误...

java.lang.IllegalArgumentException: Limit of total fields [1000] in index [shopfront] has been exceeded
at org.elasticsearch.index.mapper.MapperService.checkTotalFieldsLimit(MapperService.java:482) ~[elasticsearch-5.0.1.jar:5.0.1]
at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:343) ~[elasticsearch-5.0.1.jar:5.0.1]
at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:277) ~[elasticsearch-5.0.1.jar:5.0.1]
at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.applyRequest(MetaDataMappingService.java:323) ~[elasticsearch-5.0.1.jar:5.0.1]
at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.execute(MetaDataMappingService.java:241) ~[elasticsearch-5.0.1.jar:5.0.1]
at org.elasticsearch.cluster.service.ClusterService.runTasksForExecutor(ClusterService.java:555) ~[elasticsearch-5.0.1.jar:5.0.1]
at org.elasticsearch.cluster.service.ClusterService$UpdateTask.run(ClusterService.java:896) ~[elasticsearch-5.0.1.jar:5.0.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:451) ~[elasticsearch-5.0.1.jar:5.0.1]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:238) ~[elasticsearch-5.0.1.jar:5.0.1]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:201) ~[elasticsearch-5.0.1.jar:5.0.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]

我们使用PHP作为elasticsearch客户端来进行从MySQL到Elastic的批量上传.在做了一些谷歌搜索之后我得到了这条信息 - https://discuss.elastic.co/t/es-2-3-5-x-metricbeat-index-field-limit/66821

在某处我还读到使用"index.mapping.total_fields.limit"将解决这个问题.但是,无法理解如何在我的PHP代码中使用它.这是我的PHP代码.

$params = ['body' => []];

$i = 1;
foreach ($productsList as $key => $value) {

    $params['body'][] = [
        'index' => [
            '_index' => 'shopfront',
            '_type' => 'products'
        ],
        'settings' => ['index.mapping.total_fields.limit' => 3000]
    ];

    $params['body'][] = [
        'product_displayname' => $value['product_displayname'],
        'product_price' => $value['product_price'],
        'popularity' => $value['popularity'],
        'lowestcomp_price' => $value['lowestcomp_price']
    ];

    // Every 1000 documents stop and send the bulk request
    if ($i % 1000 == 0) {
        $responses = $client->bulk($params);

        // erase the old bulk request
        $params = ['body' => []];

        // unset the bulk response when you are done to save memory
        unset($responses);
    }

    $i++;
}

// Send the last batch if it exists
if (!empty($params['body'])) {
    $responses = $client->bulk($params);
}

注意 - 我在Elasticsearch 2.4.1中使用了相同的代码,它的工作正常.

Answer 1

在ES 5中,ES人员决定限制映射类型可以包含的字段数,以防止映射爆炸.正如您所注意到的,该限制已设置为每个映射1000个字段,但您可以通过index.mapping.total_fields.limit在索引创建时指定设置或通过更新索引设置来提升该限制以满足您的需要,如下所示:

curl -XPUT 'localhost:9200/shopfront/_settings' -d '
{
    "index.mapping.total_fields.limit": 3000
}'

请注意,您还需要问问自己是否拥有这么多字段是件好事.你需要它们吗？你可以结合一些吗？等等