HttpWebRequest - 我可以在多个线程的同时进行多次调用

Question

我使用HttpWebRequest来创建网页请求,而不是解析它们.

HttpWebRequest request = (HttpWebRequest)WebRequest.Create(address);

然后如果更多线程调用

HttpWebResponse response = (HttpWebResponse)request.GetResponse()

同时,每个应该得到它自己的响应还是线程2有可能获得thread7的响应？

Obs:所有线程的地址都相同,只有POST参数发生变化

 public class CheckHelper
{
    public  string GetPOSTWebsiteResponse(string WebAddress, string year)
    {
        StringBuilder QuerryData = new StringBuilder();
        String ResponseString;
        QuerryData.Append("forYear"+ "=" + year);

        #region build request
        HttpWebRequest request = (HttpWebRequest)WebRequest.Create(WebAddress);
        // Set the Method property of the request to POST
        request.Method = "POST";

        NameValueCollection headers = request.Headers;
        Type t = headers.GetType();
        PropertyInfo p = t.GetProperty("IsReadOnly", BindingFlags.Instance | BindingFlags.IgnoreCase | BindingFlags.NonPublic | BindingFlags.FlattenHierarchy);
        p.SetValue(headers, false, null);
        byte[] byteArray = Encoding.UTF8.GetBytes(QuerryData.ToString());
        request.ContentType = "application/x-www-form-urlencoded";
        request.ContentLength = byteArray.Length;

        #endregion

        // Get the request stream.
        using (Stream requestStream = request.GetRequestStream())
        {
            // Write the data to the request stream.
            requestStream.Write(byteArray, 0, byteArray.Length);
            // Close the Stream object.
        }



        #region get response
        using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
        {
            //Get the stream containing content returned by the server.
            using (var responseStream = response.GetResponseStream())
            {
                // Open the stream using a StreamReader for easy access.
                using (StreamReader responseReader = new StreamReader(responseStream))
                {
                    // Read the content.
                    ResponseString = responseReader.ReadToEnd();


                }
            }
        }

        #endregion        
        return ResponseString;
      }
}

这是我如何使用该方法:

            Dictionary<int, Thread> threads=new Dictionary<int,Thread>();
            foreach (var year in AvailableYears)
            {
                threads[year] = new Thread(delegate()
                    {
                    var client=new CheckHelper(); 
                    string response=client.GetPOSTWebsiteResponse("http://abc123.com", year.ToString())
                    //The thread for year 2003 may get the response for the year 2007

                    responsesDictionary[year]=response;
                    });
                threads[year].Start();

            }
            //this is to force the main thread to wait until all responses are    received:
        foreach(var th in threads.Values){
                th.Join(10000);
            }

请告诉我我错在哪里？我该如何更改代码？请帮忙,我在网上找不到任何有用的东西!

Answer 1

说实话,我不相信你试图做的多线程会让你获得任何性能提升.并且看到在这里创建的线程数没有阈值,存在比单线程(顺序)操作更糟糕的性能.

理想情况是您有异步工作流程.你的循环是这样的:

GetAsyncRequest MakeAsyncRequest ReceiveResponseAsync ProcessResponse WaitForAllRequestProcessingToComplete(可选)

这样每个步骤的结果都会输入下一个(如果有结果)和下一步.并且您在收到响应后立即处理响应,而不是在继续处理响应之前累积(加入/阻止)所有响应.使用.NET 4.0中的Tasks和ContinueWith可以很容易地完成这种事情,并且看到你使用.NET 4.0我强烈建议你按照上面的描述进行操作.

但是,如果您无法将处理转换为异步工作流,那么......

下面显示的方法是调用Url并返回响应的方法.该方法使用异步调用但阻塞,因为您的设计似乎是这样.

static string GetWebResponse(string url, NameValueCollection parameters)
{
  var httpWebRequest = (HttpWebRequest)WebRequest.Create(url);
  httpWebRequest.ContentType = "application/x-www-form-urlencoded";
  httpWebRequest.Method = "POST";

  var sb = new StringBuilder();
  foreach (var key in parameters.AllKeys)
    sb.Append(key + "=" + parameters[key] + "&");
  sb.Length = sb.Length - 1;

  byte[] requestBytes = Encoding.UTF8.GetBytes(sb.ToString());
  httpWebRequest.ContentLength = requestBytes.Length;

  using (var requestStream = httpWebRequest.GetRequestStream())
  {
    requestStream.Write(requestBytes, 0, requestBytes.Length);
    requestStream.Close();
  }

  Task<WebResponse> responseTask = Task.Factory.FromAsync<WebResponse>(httpWebRequest.BeginGetResponse, httpWebRequest.EndGetResponse, null);
  using (var responseStream = responseTask.Result.GetResponseStream())
  {
    var reader = new StreamReader(responseStream);
    return reader.ReadToEnd();
  }
}

你这样称呼它:

  ServicePointManager.DefaultConnectionLimit = 20;//Please test different numbers here
  var tasks = new List<Task<string>>();
  for (int i = 1990; i < 2090; i++)
  {
    var postParameters = new NameValueCollection();
    postParameters.Add("data", i.ToString());
    tasks.Add(Task.Factory.StartNew(() => { return GetWebResponse("http://www.abc123.com", postParameters); }));
  }
  Task.WaitAll(tasks.ToArray());
  //At this point tasks[0].Result will be the result (The Response) of the first task
  //tasks[1].Result will be the result of the second task and so on.

看看这是否适合你.

如果您确实需要多线程功能,当然只要看到您只能访问一个站点,就必须衡量性能优势,因为站点需要能够处理请求的冲击,并且客户端需要创建成本线程,只做一些I/O绑定任务可能最终成本太高,最终没有性能提升.

此外,如果不调整ServicePointManager中的DefaultConnectionLimit,您将永远不会获得超过2个线程,因为您要攻击一个域,默认限制是每个域2个线程.

我坚持使用我提供的代码,如果只存在性能问题,那么我会以其他方式来看待它.

编辑: 使用Async I/O时,您不是使用工作线程而是使用I/O线程.所以基本上你不想使用QueueUserWorkItem(创建线程)或者你不是自己创建线程.

我提出的代码是使用异步I/O,如果要尽可能快地同时执行多个请求.

for循环(在第二个代码清单中)几乎立即完成,即使在示例中它循环100次迭代,然后等待所有I/O请求完成.ThreadPool和OS将尽快并尽快处理I/O作业的运行.实际上因为这些作业是I/O绑定的,所以你也不会看到你的CPU利用率上升(除非你以后正在进行cpu绑定工作).

只需使用ServiceManager.DefaultConnectionLimit即可在需要时获得更多加速.请注意,这也会影响服务(服务器),因为如果您同时发出大量请求,那么您正在调用的服务器负载很重,而这可能不是您想要的.所以这是你需要达成的平衡.

在调用Task.WaitAll之后,您可以迭代任务集合,并使用代码清单中注释行中显示的语法获取每个任务的结果.