HttpClient和ASP.NET Core 2.0 Web服务之间的连接已关闭错误

Question

我有一个在IIS上运行的ASP.NET Core 2.0 Web服务.控制器的一种方法或多或少看起来像这样:

[HttpGet()]
public IActionResult Test()
{
    // do some db updates and get data
    var result = DoSomeStuff();
    // serialize data to byte array
    var output = Serialize(result);

    return File(output, "application/octet-stream");
}

它执行一些数据库更新,从表中查询记录,序列化数据并将其作为响应发送.数据以二进制格式发送.我正在使用MessagePack-CSharp作为序列化程序.

然后我有客户端应用程序与此Web服务进行通信.它是.NET Standard 2.0库,从.NET 4.6.1控制台应用程序引用.我HttpClient用于请求和HttpResponseMessage.Content.ReadAsByteArrayAsync()阅读响应(具体代码见下文).

我想做一些测试.我的桌子有cca.80列,包含cca.140000条记录.所有这些都应该发送给客户.从db获取数据需要几秒钟,然后它是序列化的所有内容和cca的结果.34MB发送给客户端.

我有10个客户.当他们连续调用webservice时,一切正常.当我强调web服务和并行启动客户端时,我几乎总是会遇到一些错误(通常有一两个失败,有时甚至是4-5).

以下是例外情况,它是从ReadAsByteArrayAsync电话中提出的:

System.AggregateException: One or more errors occurred. ---> System.Net.Http.HttpRequestException: Error while copying content to a stream. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
   at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)
   at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
   --- End of inner exception stack trace ---
   at System.Net.ConnectStream.EndRead(IAsyncResult asyncResult)
   at System.IO.Stream.<>c.<BeginEndReadAsync>b__43_1(Stream stream, IAsyncResult asyncResult)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncTrimPromise`1.Complete(TInstance thisRef, Func`3 endMethod, IAsyncResult asyncResult, Boolean requiresSynchronization)
...
---> (Inner Exception #0) System.Net.Http.HttpRequestException: Error while copying content to a stream. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
   at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)
   at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
   --- End of inner exception stack trace ---
   at System.Net.ConnectStream.EndRead(IAsyncResult asyncResult)
   at System.IO.Stream.<>c.<BeginEndReadAsync>b__43_1(Stream stream, IAsyncResult asyncResult)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncTrimPromise`1.Complete(TInstance thisRef, Func`3 endMethod, IAsyncResult asyncResult, Boolean requiresSynchronization)
...

我找到了几个与这种异常相关的SO线程(例如这里),所以我最初认为这是一个与客户端相关的问题.答案建议:

• 切换到HTTP 1.0
• 设置Connection: close而不是Connection: keep-alive
• 反之亦然

没有什么对我有用.我想我在某处看到HttpClient中有一些错误(现在找不到源代码).我试图使用System.Net.HttpNuget的最新软件包.同样的问题.我创建了.NET Core控制台应用程序并使用了Core版本HttpClient.同样的问题.我用HttpWebRequest而不是HttpClient.同样的根本问题.

我在同一台VM机器上运行webservice和客户端.为了排除一些本地问题,我从其他计算机同时运行客户端.同样的问题.

所以我最终得到了简化的代码(只有一个应用程序有10个线程):

private async void Test_Click(object sender, RoutedEventArgs e)
{
    try
    {
        var tasks = Enumerable.Range(1, 10).Select(async i => await Task.Run(async () => await GetContent(i))).ToList();

        await Task.WhenAll(tasks);

        MessageBox.Show(String.Join(Environment.NewLine, tasks.Select(t => t.Result.ToString())));
    }
    catch (Exception ex)
    {
        MessageBox.Show(ex.ToString());
    }
}

private async Task<Int32> GetContent(Int32 id)
{
    using (var httpClient = new HttpClient())
    {
        var url = "http://localhost/TestService/api/test";

        using (var responseMessage = await httpClient.GetAsync(url).ConfigureAwait(false))
        {
            // just read everything and return length
            // ReadAsByteArrayAsync throws sometimes an exception
            var content = await responseMessage.Content.ReadAsByteArrayAsync();
            return content.Length;
        }
    }
}

我对实际的流量感到好奇,所以我设置了Fiddler.当错误发生时,Fiddler显示响应确实已损坏,并且实际上只发送了部分假设数据量(6MB,20MB,...而不是34MB).好像它是随机中断的.我和Wireshark玩了一段时间,我看到RST/ACK数据包是从服务器发送的,但我对分析这种低级通信还不够好.

所以,我专注于服务器端.当然,我仔细检查控制器的方法是否有任何异常.一切正常.我将日志级别设置为跟踪,并在日志中找到以下内容:

info: Microsoft.AspNetCore.Server.Kestrel[28]
      Connection id "0HL89D9NUNEOQ", Request id "0HL89D9NUNEOQ:00000001": the connection was closed becuase the response was not read by the client at the specified minimum data rate.
dbug: Microsoft.AspNetCore.Server.Kestrel[10]
      Connection id "0HL89D9NUNEOQ" disconnecting.
...
info: Microsoft.AspNetCore.Server.Kestrel.Transport.Libuv[14]
      Connection id "0HL89D9NUNEOQ" communication error.
Microsoft.AspNetCore.Server.Kestrel.Transport.Libuv.Internal.Networking.UvException: Error -4081 ECANCELED operation canceled

我没有发现任何有趣的和ASP.NET Core特定的相关的错误.根据此文档,IIS可以选择在向客户端发送响应时指定最小吞吐率,具有以下设置:

<system.applicationHost>
  <webLimits minBytesPerSecond="0"/>
</system.applicationHost>

我在我的中使用它Web.config,但它没有效果(它是应用于ASP.NET核心应用程序还是只有完整的框架设置？).

我试图返回FileStreamResult而不是FileContentResult,但再次 - 它没有帮助.

与客户端类似,我也尝试为服务器端找到最小的可重现代码.方法刚刚Thread.Sleep(8000)(而不是db调用),然后生成随机50Mb字节数组并返回它.这没有任何问题,所以我想我会继续调查这个方向.我知道db可能是这里的瓶颈,但不确定它是如何导致这种情况的(没有超时异常,没有死锁,......).

有什么建议吗？我至少想知道它是服务器还是真正与客户端相关的问题.

Answer 1

看起来您的吞吐量低于最低数据速率.Kestrel Fundamentals中描述了此行为:

如果数据以指定的速率(以字节/秒为单位)进入,则Kestrel会每秒检查一次.如果速率低于最小值,则连接超时.宽限期是Kestrel给客户端将发送速率提高到最小的时间; 在此期间不检查费率.宽限期有助于避免丢弃由于TCP慢启动而最初以低速率发送数据的连接.

默认最小速率为240字节/秒,宽限期为5秒.

最低费率也适用于响应.设置请求限制和响应限制的代码是除了具有相同RequestBody或Response在属性和接口名称.

您可以在Program.cs中配置它,如下所示:

var host = new WebHostBuilder() 
    .UseKestrel(options => 
    { 
        options.Limits.MinResponseDataRate = null;
    })

设置此选项null表示不应强制执行最低数据速率.