的NodeJS.处理 字符编码

Question

我在处理字符编码方面遇到了困难.我正在尝试抓取以下网址:

http://www.google.com/movies?near=Montreal&date=0

我的代码看起来像这样:

var http = require('http');
var url = require('url');
var Iconv  = require('iconv').Iconv;

var location = 'montreal';

var googleMovies = url.parse("http://www.google.com/movies?near=" + location);

var req = http.request(googleMovies, function(response) {
    var str = '';
    response.on('data', function(chunk) {
        str += chunk;
    });
    response.on('end', function() {

        var iconv = new Iconv('latin1', 'UTF-8');
        str = iconv.convert(str).toString();

        console.log(str);
    });
});
req.end()

我第一次尝试没有:

    var iconv = new Iconv('latin1', 'UTF-8');
    str = iconv.convert(str).toString();

但这导致了 角色.

我在本页测试了上面列出的来源:

http://nlp.fi.muni.cz/projects/chared/

它似乎将它视为latin1,但事情可能是错误的.

Answer 1

字符来自串联:

response.on('data', function(chunk) {
    str += chunk;
});

这会将每个转换为默认值为chunk a的a .s中任何无效为UTF-8的序列都将丢失,并在此时替换为 .Stringencodingutf8Buffer

你要留下chunks作为Buffers直到之后convert().它们可以收集Array在一起并与之结合使用Buffer.concat().

var chunks = [];

response.on('data', function (chunk) {
    chunks.push(chunk);
});

response.on('end', function () {
    var iconv = new Iconv('latin1', 'UTF-8');
    var str = iconv.convert(Buffer.concat(chunks)).toString();
    console.log(str);
});