使用System.Net.Http.HttpClient进行并行HTTP请求

Question

我正试图找出使用Task和并行化HTTP请求的正确方法async/await.我正在使用HttpClient已经具有异步方法来检索数据的类.如果我只是在foreach循环中调用它并等待响应,则一次只发送一个请求(这是有道理的,因为在await控制期间,控制返回到我们的事件循环,而不是foreach循环的下一次迭代).

我的包装HttpClient看起来像这样

public sealed class RestClient
{
    private readonly HttpClient client;

    public RestClient(string baseUrl)
    {
        var baseUri = new Uri(baseUrl);

        client = new HttpClient
        {
            BaseAddress = baseUri
        };
    }

    public async Task<Stream> GetResponseStreamAsync(string uri)
    {
        var resp = await GetResponseAsync(uri);
        return await resp.Content.ReadAsStreamAsync();
    }

    public async Task<HttpResponseMessage> GetResponseAsync(string uri)
    {
        var resp = await client.GetAsync(uri);
        if (!resp.IsSuccessStatusCode)
        {
            // ...
        }

        return resp;
    }

    public async Task<T> GetResponseObjectAsync<T>(string uri)
    {
        using (var responseStream = await GetResponseStreamAsync(uri))
        using (var sr = new StreamReader(responseStream))
        using (var jr = new JsonTextReader(sr))
        {
            var serializer = new JsonSerializer {NullValueHandling = NullValueHandling.Ignore};
            return serializer.Deserialize<T>(jr);
        }
    }

    public async Task<string> GetResponseString(string uri)
    {
        using (var resp = await GetResponseStreamAsync(uri))
        using (var sr = new StreamReader(resp))
        {
            return sr.ReadToEnd();
        }
    }
}

我们的事件循环调用的代码是

public async void DoWork(Action<bool> onComplete)
{
    try
    {
        var restClient = new RestClient("https://example.com");

        var ids = await restClient.GetResponseObjectAsync<IdListResponse>("/ids").Ids;

        Log.Info("Downloading {0:D} items", ids.Count);
        using (var fs = new FileStream(@"C:\test.json", FileMode.Create, FileAccess.Write, FileShare.Read))
        using (var sw = new StreamWriter(fs))
        {
            sw.Write("[");

            var first = true;
            var numCompleted = 0;
            foreach (var id in ids)
            {
                Log.Info("Downloading item {0:D}, completed {1:D}", id, numCompleted);
                numCompleted += 1;
                try
                {
                    var str = await restClient.GetResponseString($"/info/{id}");
                    if (!first)
                    {
                        sw.Write(",");
                    }

                    sw.Write(str);

                    first = false;
                }
                catch (HttpException e)
                {
                    if (e.StatusCode == HttpStatusCode.Forbidden)
                    {
                        Log.Warn(e.ResponseMessage);
                    }
                    else
                    {
                        throw;
                    }
                }
            }

            sw.Write("]");
        }

        onComplete(true);
    }
    catch (Exception e)
    {
        Log.Error(e);
        onComplete(false);
    }
}

我试过的,涉及不同的方法了一把Parallel.ForEach,Linq.AsParallel和在包装循环的全部内容Task.

Answer 1

基本思想是跟踪所有异步任务,并立即等待它们.最简单的方法是将foreach的主体提取为单独的异步方法,并执行以下操作:

var tasks = ids.Select(i => DoWorkAsync(i));
await Task.WhenAll(tasks);

这样,各个任务将单独发布(仍然按顺序发布,但不等待I/O完成),并且您可以同时等待所有任务.

请注意,您还需要进行一些配置 - 默认情况下,HTTP会受到限制,只允许两个同时连接到同一服务器.