从Python 2移植到Python 3:'utf-8编解码器无法解码字节'

Question

嘿,我试图将这个小片段从2端口移植到Python 3.

Python 2:

def _download_database(self, url):
  try:
    with closing(urllib.urlopen(url)) as u:
      return StringIO(u.read())
  except IOError:
    self.__show_exception(sys.exc_info())
  return None

Python 3:

def _download_database(self, url):
  try:
    with closing(urllib.request.urlopen(url)) as u:
      response = u.read().decode('utf-8')
      return StringIO(response)
  except IOError:
    self.__show_exception(sys.exc_info())
  return None

但我还是得到了

utf-8 codec can't decode byte 0x8f in position 12: invalid start byte

我需要使用StringIO,因为它是一个zipfile,我想用该函数解析它:

   def _parse_zip(self, raw_zip):
  try:
     zip = zipfile.ZipFile(raw_zip)

     filelist = map(lambda x: x.filename, zip.filelist)
     db_file  = 'IpToCountry.csv' if 'IpToCountry.csv' in filelist else filelist[0]

     with closing(StringIO(zip.read(db_file))) as raw_database:
        return_val = self.___parse_database(raw_database)

     if return_val:
        self._load_data()

  except:
     self.__show_exception(sys.exc_info())
     return_val = False

  return return_val

raw_zip是download_database func的返回值

Answer 1

utf-8无法解码任意二进制数据.

utf-8是一种字符编码,可用于将文本(例如,str在Python 3中表示为类型 - Unicode代码点序列)编码为字节串(bytes类型 - 字节序列([0,255]中的小整数) interval))并将其解码回来.

utf-8不是唯一的字符编码.有些字符编码与utf-8不兼容.即使.decode('utf-8')没有提出异常; 这并不意味着结果是正确的 - 如果你使用错误的字符编码来解码文本,你可能会得到mojibake.请参阅在Python中获取HTTP响应的字符集/编码的好方法.

您的输入是zip文件 - 二进制数据不是文本,因此您不应尝试将其解码为文本.

Python 3可帮助您查找与混合二进制数据和文本相关的错误.要将代码从Python 2移植到Python 3,您应该了解文本(Unicode)与二进制数据(字节)的区别.

strPython 2上的字节串可用于二进制数据和(编码)文本.除非from __future__ import unicode_literals存在; ''literal在Python中创建一个bytestring.u''创建unicode实例.在Python 3上str类型是Unicode.bytes指的是Python 3和Python 2.7上的字节序列(bytes是strPython 2上的别名).在Python 2/3上b''创建bytes实例.

urllib.request.urlopen(url)返回类似文件的对象(二进制文件),您可以在某些情况下按原样传递它,例如,即时解码远程gzip压缩内容:

#!/usr/bin/env python3
import xml.etree.ElementTree as etree
from gzip import GzipFile
from urllib.request import urlopen, Request

with urlopen(Request("http://smarkets.s3.amazonaws.com/oddsfeed.xml",
                     headers={"Accept-Encoding": "gzip"})) as response, \
     GzipFile(fileobj=response) as xml_file:
    for elem in getelements(xml_file, 'interesting_tag'):
        process(elem)

ZipFile()需要一个seek()文件,因此你无法urlopen()直接传递.您必须先下载内容.你可以用io.BytesIO()它来包装它:

#!/usr/bin/env python3
import io
import zipfile
from urllib.request import urlopen

url = "http://www.pythonchallenge.com/pc/def/channel.zip"
with urlopen(url) as r, zipfile.ZipFile(io.BytesIO(r.read())) as archive:
    print({member.filename: archive.read(member) for member in archive.infolist()})

StringIO()是文本文件.它在Python 3中存储Unicode.