AppFabric扩展问题疑难解答(间歇性错误代码<ERRCA0017>:SubStatus <ES0006>错误)

Question

我们已经为我们的Web应用程序实现了AppFabric Windows Server Cache.最初,我们能够毫无问题地使用缓存.然后我们将流量增加了大约100倍,并开始出现间歇性异常.例外情况大约每2天发生一次,大约每分钟发生一次.

我们的配置:

• 9个Web服务器在缓存中插入/检索对象:
  • 主要是临时500字节操作类型对象
  • 使用1个命名区域
  • 存储有标签的对象
  • 已批量检索给定标签
• 缓存集群:
  • 1个主机(主管)AppFabric 1.1(get-cachehost报告的版本为3)
  • SQL配置提供程序
  • 主机上有96GB的RAM,默认50%(48GB)分配给AppFabric
  • 缓存主机配置
  • 缓存客户端配置

错误按顺序发生(在1分钟内,九个Web服务器中的每一个都会发生异常):

• System.Net.Sockets.SocketException:远程主机Microsoft.ApplicationServer.Caching.DataCacheException强制关闭现有连接: ErrorCode<ERRCA0016>:SubStatus<ES0001>:The connection was terminated, possibly due to server or network problems or serialized Object size is greater than MaxBufferSize on server. Result of the request is unknown. ---> System.ServiceModel.CommunicationException: The socket connection was aborted. This could be caused by an error processing your message or a receive timeout being exceeded by the remote host, or an underlying network resource issue. Local socket timeout was '00:15:00'. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host --- End of inner exception stack trace --- at System.Runtime.AsyncResult.End[TAsyncResult](IAsyncResult result) at System.ServiceModel.Channels.FramingDuplexSessionChannel.EndReceive(IAsyncResult result) at Microsoft.ApplicationServer.Caching.WcfClientChannel.CompleteProcessing(IAsyncResult result) --- End of inner exception stack trace --- at Microsoft.ApplicationServer.Caching.DataCache.ThrowException(ResponseBody respBody) at Microsoft.ApplicationServer.Caching.DataCache.GetNextBatch(String region, DataCacheTag[] tags, GetByTagsOperation op, IMonitoringListener listener, Byte[][]& state, Boolean& more) at Microsoft.ApplicationServer.Caching.CacheEnumerator.MoveNext() at System.Linq.Enumerable.WhereSelectEnumerableIterator'2.MoveNext() at System.Linq.Enumerable.<ExceptIterator>d__99'1.MoveNext() at System.Collections.Generic.List'1..ctor(IEnumerable'1 collection) at System.Linq.Enumerable.ToList[TSource](IEnumerable'1 source)

• Microsoft.ApplicationServer.Caching.DataCacheException: ErrorCode<ERRCA0017>:SubStatus<ES0006>:There is a temporary failure. Please retry later. (One or more specified cache servers are unavailable, which could be caused by busy network or servers. For on-premises cache clusters, also verify the following conditions. Ensure that security permission has been granted for this client account, and check that the AppFabric Caching Service is allowed through the firewall on all cache hosts. Also the MaxBufferSize on the server must be greater than or equal to the serialized object size sent from the client.) at Microsoft.ApplicationServer.Caching.DataCache.ThrowException(ResponseBody respBody) at Microsoft.ApplicationServer.Caching.DataCache.GetNextBatch(String region, DataCacheTag[] tags, GetByTagsOperation op, IMonitoringListener listener, Byte[][]& state, Boolean& more) at Microsoft.ApplicationServer.Caching.CacheEnumerator.MoveNext() at System.Linq.Enumerable.WhereSelectEnumerableIterator'2.MoveNext() at System.Linq.Enumerable.<ExceptIterator>d__99'1.MoveNext() at System.Collections.Generic.List'1..ctor(IEnumerable'1 collection) at System.Linq.Enumerable.ToList[TSource](IEnumerable'1 source)

• Microsoft.ApplicationServer.Caching.DataCacheException: ErrorCode<ERRCA0018>:SubStatus<ES0001>:The request timed out. at Microsoft.ApplicationServer.Caching.DataCache.ThrowException(ResponseBody respBody) at Microsoft.ApplicationServer.Caching.DataCache.GetNextBatch(String region, DataCacheTag[] tags, GetByTagsOperation op, IMonitoringListener listener, Byte[][]& state, Boolean& more) at Microsoft.ApplicationServer.Caching.CacheEnumerator.MoveNext() at System.Linq.Enumerable.WhereSelectEnumerableIterator'2.MoveNext() at System.Linq.Enumerable.<ExceptIterator>d__99'1.MoveNext() at System.Collections.Generic.List'1..ctor(IEnumerable'1 collection) at System.Linq.Enumerable.ToList[TSource](IEnumerable'1 source)

我们还在缓存服务器上创建了一个tracelog会话,以捕获更多信息来诊断问题 - 任何有关如何分析此问题的建议都将受到赞赏(如果需要,我可以提供此建议).

我们还监控了各种AppFabric,CLR和网络性能计数器,下面是事件的屏幕截图:

提前感谢您就解决此问题可以分享的任何想法或建议.

更新1

以下是在间歇性错误(从tracelogs中抽象)期间在AppFabric缓存服务器上连续发生的异常:

• System.ServiceModel.CommunicationException: The socket connection was aborted because an asynchronous send to the socket did not complete within the allotted timeout of 00:00:00.0082078. The time allotted to this operation may have been a portion of a longer timeout. ---> System.ObjectDisposedException: The socket connection has been disposed. Object name: 'System.ServiceModel.Channels.SocketConnection'. --- End of inner exception stack trace --- at System.ServiceModel.Channels.SocketConnection.ThrowIfNotOpen() at System.ServiceModel.Channels.SocketConnection.BeginRead(Int32 offset, Int32 size, TimeSpan timeout, WaitCallback callback, Object state) at System.ServiceModel.Channels.SessionConnectionReader.BeginReceive(TimeSpan timeout, WaitCallback callback, Object state) at System.ServiceModel.Channels.SynchronizedMessageSource.ReceiveAsyncResult.PerformOperation(TimeSpan timeout) at System.ServiceModel.Channels.SynchronizedMessageSource.SynchronizedAsyncResult'1..ctor(SynchronizedMessageSource syncSource, TimeSpan timeout, AsyncCallback callback, Object state) at System.ServiceModel.Channels.FramingDuplexSessionChannel.BeginReceive(TimeSpan timeout, AsyncCallback callback, Object state) at Microsoft.ApplicationServer.Caching.WcfServerChannel.CompleteProcessing(IAsyncResult result)

• System.ServiceModel.CommunicationObjectAbortedException: The communication object, System.ServiceModel.Channels.ServerSessionPreambleConnectionReader+ServerFramingDuplexSessionChannel, cannot be used for communication because it has been Aborted. at System.Runtime.AsyncResult.End[TAsyncResult](IAsyncResult result) at System.ServiceModel.Channels.FramingDuplexSessionChannel.OnEndSend(IAsyncResult result) at Microsoft.ApplicationServer.Caching.ReplyContext.EndSend(IAsyncResult result)

• System.ServiceModel.CommunicationObjectFaultedException: The communication object, System.ServiceModel.Channels.ServerSessionPreambleConnectionReader+ServerFramingDuplexSessionChannel, cannot be used for communication because it is in the Faulted state. at System.ServiceModel.Channels.CommunicationObject.ThrowIfDisposedOrNotOpen() at System.ServiceModel.Channels.OutputChannel.Send(Message message, TimeSpan timeout) at Microsoft.ApplicationServer.Caching.ReplyContext.Reply(Message message, TimeSpan timeout)

• System.TimeoutException: Sending to via http://www.w3.org/2005/08/addressing/anonymous timed out after 00:00:15. The time allotted to this operation may have been a portion of a longer timeout. ---> System.TimeoutException: Cannot claim lock within the allotted timeout of 00:00:15. The time allotted to this operation may have been a portion of a longer timeout. --- End of inner exception stack trace --- at System.ServiceModel.Channels.FramingDuplexSessionChannel.OnSend(Message message, TimeSpan timeout) at System.ServiceModel.Channels.OutputChannel.Send(Message message, TimeSpan timeout) at Microsoft.ApplicationServer.Caching.ReplyContext.Reply(Message message, TimeSpan timeout)

更新2

经过一天的故障排除后,我们采取了以下措施,这些措

1. 基于这个和这个,我们增加了maxConnectionsToServer对3.结果,AppFabric Caching:Cache perf计数器记录的客户端请求数增加了50%,但间歇性错误并未停止

2. 我们在Cache Server配置上增加了to maxBufferSize和maxBufferPoolSizeto 2147483647(int32.max).到目前为止,我们能够处理300x流量没有错误.我们将继续增加交通量和监控.更多要关注的更新

更新3

我们向群集添加了另外两台主机,每台16GB,并启用了HighAvailability模式(via Secondaries=1).目前,原始主机仍然在96GB RAM的集群中 - 所有主机都有cacheSize = 12GB.在缓存客户端上,我们将MaxConnectionToServerto 增加12(每个核心1个).以下是我们的发现:

• 偶尔我们得到(每10分钟一次或两次):
  • ErrorCode<ERRCA0017>:SubStatus<ES0005>:There is a temporary failure. Please retry later. (There was a contention on the store.)
  • ErrorCode<ERRCA0017>:SubStatus<ES0004>:There is a temporary failure. Please retry later. (Replication queue was full. This may happen during reconfiguration of cluster hosts.)
• 如上所述,原始的96​​GB缓存主机仍然会经历1分钟的中断.新的缓存主机没有经历过中断

我们计划从原始缓存主机中删除80GB内存.更多要关注的更新.

更新4

这个问题似乎已经通过将缓存主机中的RAM数量减少到16GB来解决.我们不再看到流量增加到400倍的间歇性错误.似乎是封闭的.现在转到下一期:高可用性

Answer 1

重新发布Jeff-ITGuy在Social.msdn.microsoft.com上给出的答案：

您似乎遇到了与我目前与 Microsoft 合作的问题几乎相同的问题。如果是同样的问题，很可能是GC时间过长导致AppFabric的响应时间延迟造成的。从你的性能计数器来看，当你开始遇到问题时，GC 时间似乎突然增加，所以这可能是同一个问题。

Microsoft 正在积极调查此问题。同时，为了缓解问题（至少从我们的发现来看），您可以使用更少的内存运行更多的服务器（缩小 GC 所占用的内存空间的大小），并且可以增加客户端上的 RequestTimeout。默认情况下设置为 15000（15 秒），但我们尝试将其提高到 30000，这有助于消除一些问题。在我看来，这不是一个好的长期解决方案，只是传递信息。我已经看到服务器只有 24GB 内存（12GB 用于缓存）的问题，当我们尝试使用 8GB 服务器并将 4GB 设置为缓存时，情况才真正明显好转 - 这导致 GC 做得更好。

希望这会有所帮助，但如果这是问题，我认为目前还没有解决方案。

它确实有帮助，在我们将缓存主机 RAM 减少到 16GB 后，间歇性错误就停止了。