org.apache.http.ContentTooLongException：对于配置的缓冲区限制，实体内容太长 [105539255] [104857600]

Question

我正在尝试从我的索引 (ElasticSearch) 中获取索引的 PDF 文档。我已经使用摄取附件处理器插件索引了我的 pdf 文档。共有 2500 份文档与 PDF 附件一起被索引。

现在我通过搜索 PDF 的内容来获取这些 PDF，并得到以下错误。

org.apache.http.ContentTooLongException: entity content is too long [105539255] for the configured buffer limit [104857600]
    at org.elasticsearch.client.HeapBufferedAsyncResponseConsumer.onEntityEnclosed(HeapBufferedAsyncResponseConsumer.java:76)
    at org.apache.http.nio.protocol.AbstractAsyncResponseConsumer.responseReceived(AbstractAsyncResponseConsumer.java:131)
    at org.apache.http.impl.nio.client.MainClientExec.responseReceived(MainClientExec.java:315)
    at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseReceived(DefaultClientExchangeHandlerImpl.java:147)
    at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.responseReceived(HttpAsyncRequestExecutor.java:303)
    at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:255)
    at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
    at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
    at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)
    at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
    at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
    at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
    at java.lang.Thread.run(Thread.java:748)
Exception in thread "main" java.lang.NullPointerException
    at com.es.utility.DocumentSearch.main(DocumentSearch.java:88)

请找到我的 Java API 代码以从 ElasticSearch 中获取文档

private final static String ATTACHMENT = "document_attachment";
private final static String TYPE = "doc";

public static void main(String args[])
{
    RestHighLevelClient restHighLevelClient = null;

    try {
        restHighLevelClient = new RestHighLevelClient(RestClient.builder(new HttpHost("localhost", 9200, "http"),
                new HttpHost("localhost", 9201, "http")));

    } catch (Exception e) {
        System.out.println(e.getMessage());
    }



    SearchRequest contentSearchRequest = new SearchRequest(ATTACHMENT); 
    SearchSourceBuilder contentSearchSourceBuilder = new SearchSourceBuilder();
    contentSearchRequest.types(TYPE);
    QueryBuilder attachmentQB = QueryBuilders.matchQuery("attachment.content", "activa");
    contentSearchSourceBuilder.query(attachmentQB);
    contentSearchSourceBuilder.size(50);
    contentSearchRequest.source(contentSearchSourceBuilder);
    SearchResponse contentSearchResponse = null;
    System.out.println("Request --->"+contentSearchRequest.toString());
    try {
        contentSearchResponse = restHighLevelClient.search(contentSearchRequest);
    } catch (IOException e) {
        e.getLocalizedMessage();
    }

    try {
        System.out.println("Response --->"+restHighLevelClient.search(contentSearchRequest)); // am printing the mentioned error from this line.
    } catch (IOException e) {
        e.printStackTrace();
    }
    SearchHit[] contentSearchHits = contentSearchResponse.getHits().getHits();
    long contenttotalHits=contentSearchResponse.getHits().totalHits;
    System.out.println("condition Total Hits --->"+contenttotalHits);

我正在使用 ElasticSearch 版本 6.2.3

Answer 1

您需要http.max_content_length在elasticsearch.yml配置文件中增加。

默认情况下，它设置为 100MB (100*1024*1024 = 104857600)，因此您可能需要将其设置得稍高一些。

更新

这实际上是一个不同的问题，在这里解释。基本上，默认情况下会HttpAsyncResponseConsumerFactory在堆内存中缓冲整个响应主体，但默认情况下最多只能缓冲 100mb。解决方法是为该缓冲区配置另一个大小，但您唯一的选择是改用低级 REST 客户端。在 ES 7 中，您将能够使用名为 的类在高级 REST 客户端上执行此操作RequestOptions，但该类尚未发布。

long BUFFER_SIZE = 120 * 1024 * 1024;     <---- set buffer to 120MB instead of 100MB
Map<String, String> params = Collections.emptyMap();
HttpEntity entity = new NStringEntity(contentSearchSourceBuilder.toString(), ContentType.APPLICATION_JSON);
HttpAsyncResponseConsumerFactory.HeapBufferedResponseConsumerFactory consumerFactory =
        new HttpAsyncResponseConsumerFactory.HeapBufferedResponseConsumerFactory(BUFFER_SIZE);
Response response = restClient.performRequest("GET", "/document_attachment/doc/_search", params, entity, consumerFactory);