C#使用C#WebClient或HttpWebRequest将网站下载到字符串中

Question

我正在尝试下载网站的内容.但是对于某个网页,返回的字符串包含混乱的数据,包含许多 字符.

这是我最初使用的代码.

HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(url);
req.Method = "GET";
req.UserAgent = "Mozilla/5.0 (Windows; U; MSIE 9.0; WIndows NT 9.0; en-US))";
string source;
using (StreamReader reader = new StreamReader(req.GetResponse().GetResponseStream()))
{
    source = reader.ReadToEnd();
}
HtmlAgilityPack.HtmlDocument doc = new HtmlAgilityPack.HtmlDocument();
doc.LoadHtml(source);

我也尝试过使用WebClient的替代实现,但结果仍然相同:

HtmlAgilityPack.HtmlDocument doc = new HtmlAgilityPack.HtmlDocument();
using (WebClient client = new WebClient())
using (var read = client.OpenRead(url))
{
    doc.Load(read, true);
}

从搜索我想这可能是编码的问题,所以我尝试了下面发布的两个解决方案,但仍然无法使其工作.

• http://blogs.msdn.com/b/feroze_daud/archive/2004/03/30/104440.aspx
• http://bytes.com/topic/c-sharp/answers/653250-webclient-encoding

我似乎无法下载的违规网站是关于WikiPedia英文版本的United_States文章(en.wikipedia.org/wiki/United_States).虽然我已经尝试了许多其他维基百科文章,但没有看到这个问题.

Answer 1

响应是 gzip 编码的。尝试以下方法来解码流：

更新

根据 BrokenGlass 的评论，设置以下属性应该可以解决您的问题（对我有用）：

req.Headers[HttpRequestHeader.AcceptEncoding] = "gzip, deflate";
req.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip;

旧/手动解决方案：

string source;
var response = req.GetResponse();

var stream = response.GetResponseStream();
try
{
    if (response.Headers.AllKeys.Contains("Content-Encoding")
        && response.Headers["Content-Encoding"].Contains("gzip"))
    {
        stream = new System.IO.Compression.GZipStream(stream, System.IO.Compression.CompressionMode.Decompress);
    }

    using (StreamReader reader = new StreamReader(stream))
    {
        source = reader.ReadToEnd();
    }
}
finally
{
    if (stream != null)
        stream.Dispose();
}