为什么UTF-8在这种编码上失败？

Question

我即将下载以UTF-8编码的页面.所以这是我的代码:

using (WebClient client = new WebClient())
{
    client.Headers.Add("user-agent", Request.UserAgent);

    htmlPage = client.DownloadString(HttpUtility.UrlDecode(resoruce_url));

    var KeysParsed = HttpUtility.ParseQueryString(client.ResponseHeaders["Content-Type"].Replace(" ", "").Replace(";", "&"));
    var charset = ((KeysParsed["charset"] != null) ? KeysParsed["charset"] : "UTF-8");
    Response.Write(client.ResponseHeaders);

    byte[] bytePage = Encoding.GetEncoding(charset).GetBytes(htmlPage);
    using (var reader = new StreamReader(new MemoryStream(bytePage), Encoding.GetEncoding(charset)))
    {
        htmlPage = reader.ReadToEnd();
        Response.Write(htmlPage);
    }
}

所以,它设置UTF-8为编码.但是,下载的标题,例如,在我的屏幕中显示为:

Sexy cover: 60 e più di “quei dischi” vietati ai minori

而不是:

Sexy cover: 60 e più di “quei dischi” vietati ai minori

有些事情是错的,但我找不到.有任何想法吗？

Answer 1

问题是,当您获得数据时,它已经被转换.

当WebClient.DownloadString执行时,它得到的原始字节并将其转换为使用默认的编码字符串.损坏已经完成.您无法获取生成的字符串,将其重新转换为字节,然后重新解释它.

换句话说,这就是发生的事情:

// WebClient.DownloadString does, essentially, this.
byte[] rawBytes = DownloadData();
string htmlPage = Encoding.Default.GetString(rawBytes);

// Now you're doing this:
byte[] myBytes = Encoding.Utf8.GetBytes(htmlPage);

但myBytes不一定是一样的rawBytes.

如果您事先知道要使用的编码,则可以设置WebClient实例的Encoding属性.如果要根据Content-Type标头中指定的编码解释字符串,则必须下载原始字节,确定编码,并使用它来解释字符串.例如:

var rawBytes = client.DownloadData(HttpUtility.UrlDecode(resoruce_url));
var KeysParsed = HttpUtility.ParseQueryString(client.ResponseHeaders["Content-Type"].Replace(" ", "").Replace(";", "&"));
var charset = ((KeysParsed["charset"] != null) ? KeysParsed["charset"] : "UTF-8");

var theEncoding = Encoding.GetEncoding(charset);
htmlPage = theEncoding.GetString(rawBytes);