UnicodeDecodeError:'utf8'编解码器无法解码3-6位的字节:无效数据

Question

unicode是如何在python2上运行的？我只是不明白.

在这里,我从服务器下载数据并解析它为JSON.

Traceback (most recent call last):
  File "/usr/local/lib/python2.6/dist-packages/eventlet-0.9.12-py2.6.egg/eventlet/hubs/poll.py", line 92, in wait
    readers.get(fileno, noop).cb(fileno)
  File "/usr/local/lib/python2.6/dist-packages/eventlet-0.9.12-py2.6.egg/eventlet/greenthread.py", line 202, in main
    result = function(*args, **kwargs)
  File "android_suggest.py", line 60, in fetch
    suggestions = suggest(chars)
  File "android_suggest.py", line 28, in suggest
    return [i['s'] for i in json.loads(opener.open('https://market.android.com/suggest/SuggRequest?json=1&query='+s+'&hl=de&gl=DE').read())]
  File "/usr/lib/python2.6/json/__init__.py", line 307, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.6/json/decoder.py", line 319, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python2.6/json/decoder.py", line 336, in raw_decode
    obj, end = self._scanner.iterscan(s, **kw).next()
  File "/usr/lib/python2.6/json/scanner.py", line 55, in iterscan
    rval, next_pos = action(m, context)
  File "/usr/lib/python2.6/json/decoder.py", line 217, in JSONArray
    value, end = iterscan(s, idx=end, context=context).next()
  File "/usr/lib/python2.6/json/scanner.py", line 55, in iterscan
    rval, next_pos = action(m, context)
  File "/usr/lib/python2.6/json/decoder.py", line 183, in JSONObject
    value, end = iterscan(s, idx=end, context=context).next()
  File "/usr/lib/python2.6/json/scanner.py", line 55, in iterscan
    rval, next_pos = action(m, context)
  File "/usr/lib/python2.6/json/decoder.py", line 155, in JSONString
    return scanstring(match.string, match.end(), encoding, strict)
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 3-6: invalid data

谢谢!!

编辑:以下字符串导致错误:'[{"t":"q","s":"abh\xf6ren"}]'.\xf6应解码为ö(abhören)

Answer 1

您尝试解析为JSON的字符串不是以UTF-8编码的.最有可能的是它是用ISO-8859-1编码的.请尝试以下方法:

json.loads(unicode(opener.open(...), "ISO-8859-1"))

这将处理可能在JSON消息中获得的任何变音符号.

你应该阅读Joel Spolsky的绝对最低每个软件开发人员,绝对必须知道Unicode和字符集(没有借口!).我希望它能澄清你在Unicode方面遇到的一些问题.

Answer 2

我的解决方案有点滑稽.我从未想过它会像使用UTF-8编解码器一样容易保存.我使用的是notepad ++(v5.6.8).我没注意到我最初用ANSI编解码器保存了它.我正在使用单独的文件来放置所有本地化的字典.我在Notepad ++的'Encoding'选项卡下找到了我的解决方案.我选择'在没有BOM的UTF-8中编码'并保存它.它的工作非常出色.