使用Elastic Cloud/Found随机断开与主节点NoNodeAvailableException的连接

Question

我正在使用带有盾牌和传输java客户端的弹性云(以前找到).与ES通信的应用程序在heroku上运行.我正在使用一个节点对暂存环境进行压力测试

{
    "cluster_name": ...,
    "status": "yellow", 
    "timed_out": false,
    "number_of_nodes": 1,
    "number_of_data_nodes": 1,
    "active_primary_shards": 19,
    "active_shards": 19,
    "relocating_shards": 0,
    "initializing_shards": 0,
    "unassigned_shards": 7,
    "delayed_unassigned_shards": 0,
    "number_of_pending_tasks": 0,
    "number_of_in_flight_fetch": 0
}

一开始一切都很完美.但过了一段时间(3-4分钟),我开始出现一些错误.我已经将日志级别设置为跟踪,这些是我一直在犯的错误(我已经替换了...一切无关紧要的错误.

org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes were available: [[...][...][...][inet[...]]{logical_availability_zone=..., availability_zone=..., max_local_storage_nodes=1, region=..., master=true}]
    at org.elasticsearch.client.transport.TransportClientNodesService$RetryListener.onFailure(TransportClientNodesService.java:242)
    at org.elasticsearch.action.TransportActionNodeProxy$1.handleException(TransportActionNodeProxy.java:78)
    at org.elasticsearch.transport.TransportService$3.run(TransportService.java:290)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.transport.SendRequestTransportException: [...][inet[...]][indices:data/read/search]
    at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:286)
    at org.elasticsearch.shield.transport.ShieldClientTransportService.sendRequest(ShieldClientTransportService.java:41)
    at org.elasticsearch.action.TransportActionNodeProxy.execute(TransportActionNodeProxy.java:57)
    at org.elasticsearch.client.transport.support.InternalTransportClient$1.doWithNode(InternalTransportClient.java:109)
    at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:205)
    at org.elasticsearch.client.transport.support.InternalTransportClient.execute(InternalTransportClient.java:106)
    at org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:334)
    at org.elasticsearch.client.transport.TransportClient.search(TransportClient.java:416)
    at org.elasticsearch.action.search.SearchRequestBuilder.doExecute(SearchRequestBuilder.java:1122)
    at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:91)
    at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:65)
    ...
Caused by: org.elasticsearch.transport.NodeNotConnectedException: [...][inet[...]] Node not connected
    at org.elasticsearch.transport.netty.NettyTransport.nodeChannel(NettyTransport.java:936)
    at org.elasticsearch.transport.netty.NettyTransport.sendRequest(NettyTransport.java:629)
    at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:276)
    ... 

这些是我的财产

  settings = ImmutableSettings.settingsBuilder()
      .put("client.transport.nodes_sampler_interval", "5s") //Tried it with 30s, same outcome
      .put("client.transport.ping_timeout", "30s")
      .put("cluster.name", clusterName)
      .put("action.bulk.compress", false)
      .put("shield.transport.ssl", true)
      .put("request.headers.X-Found-Cluster", clusterName)
      .put("shield.user", user + ":" + password)
      .put("transport.ping_schedule", "1s") //Tried with 5s, same outcome
      .build();

我也为我的每个查询设置了:

max_query_response_size=100000
timeout_seconds=30

我在我的机器上使用ElasticSearch 1.7.2并Shield 1.3.2使用相应的(相同版本)客户端Java 1.8.0_65- Java 1.8.0_40在节点上.

我在没有压力测试的情况下得到了相同的错误,但错误发生得非常随机,所以我想重现.这就是我在单个节点中运行它的原因.

我在日志中发现了另一个错误

2016-03-07 23:35:52,177 DEBUG [elasticsearch[Vermin][transport_client_worker][T#7]{New I/O worker #16}] ssl.SslHandler (NettyInternalESLogger.java:debug(63)) - Swallowing an exception raised while writing non-app data
java.nio.channels.ClosedChannelException
    at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:433)
    at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:373)
    at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93)
    at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
    at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
    at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
    at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
    at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

热线程

0.0% (111.6micros out of 500ms) cpu usage by thread 'elasticsearch[...][transport_client_timer][T#1]{Hashed wheel timer #1}'
 10/10 snapshots sharing following 5 elements
   java.lang.Thread.sleep(Native Method)
   org.elasticsearch.common.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:445)
   org.elasticsearch.common.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:364)
   org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
   java.lang.Thread.run(Thread.java:745)

在阅读了这篇http://blog.trifork.com/2015/04/08/dealing-with-nodenotavailableexceptions-in-elasticsearch/之后,我逐渐了解了整个沟通的工作原理.我还没有测试过这个,但我相信问题就在那里.但事情是,即使我确认问题是关闭查询连接,我该如何处理？保持配置不变,只需重新连接？我禁用了keepAlive吗？如果是的话,我应该担心别的吗？

Answer 1

引用此链接： https://discuss.elastic.co/t/nonodeavailableexception-with-java-transport-client/37702作者：Konrad Beiske

您的应用程序可能会在启动时解析 IP 地址。ELB 可以随时更改 IP。为了获得最佳可靠性，您的应用程序应将 ELB 的所有 IP 添加到客户端，并定期检查 DNS 服务是否有更改。

我们的 ELB 的连接超时为 5 分钟。

以下内容应该可以帮助您修复它：

为每个请求创建一个新的 TransportClient 并不理想，因为这意味着每个请求都要进行新的连接握手，这会影响您的响应时间。如果您愿意，您可以拥有一个 TransportClient 池，但这很可能是不必要的开销，因为客户端是线程安全的。

我的建议是您创建一个小型单例服务，定期检查 DNS 服务的更改并将任何新的 IP 添加到现有的传输客户端。理论上，它可能就像每次检查时添加所有发现的 IP 一样天真，因为传输客户端将丢弃重复的地址，并清除不再可达的旧地址。