u'\ ufeff'在Python字符串中

Question

我得到了以下模式的错误:

UnicodeEncodeError: 'ascii' codec can't encode character u'\ufeff' in position 155: ordinal not in range(128)

不知道是什么u'\ufeff',它在网络抓取时显示出来.我该如何纠正这种情况？该.replace()字符串的方法不能进行这项工作.

Answer 1

Unicode字符U+FEFF是字节顺序标记或BOM,用于区分大端和小端UTF-16编码.如果使用正确的编解码器解码网页,Python将为您删除它.例子:

#!python2
#coding: utf8
u = u'ABC'
e8 = u.encode('utf-8')        # encode without BOM
e8s = u.encode('utf-8-sig')   # encode with BOM
e16 = u.encode('utf-16')      # encode with BOM
e16le = u.encode('utf-16le')  # encode without BOM
e16be = u.encode('utf-16be')  # encode without BOM
print 'utf-8     %r' % e8
print 'utf-8-sig %r' % e8s
print 'utf-16    %r' % e16
print 'utf-16le  %r' % e16le
print 'utf-16be  %r' % e16be
print
print 'utf-8  w/ BOM decoded with utf-8     %r' % e8s.decode('utf-8')
print 'utf-8  w/ BOM decoded with utf-8-sig %r' % e8s.decode('utf-8-sig')
print 'utf-16 w/ BOM decoded with utf-16    %r' % e16.decode('utf-16')
print 'utf-16 w/ BOM decoded with utf-16le  %r' % e16.decode('utf-16le')

请注意,这EF BB BF是一个UTF-8编码的BOM.它不是UTF-8所必需的,但仅作为签名(通常在Windows上).

输出:

utf-8     'ABC'
utf-8-sig '\xef\xbb\xbfABC'
utf-16    '\xff\xfeA\x00B\x00C\x00'    # Adds BOM and encodes using native processor endian-ness.
utf-16le  'A\x00B\x00C\x00'
utf-16be  '\x00A\x00B\x00C'

utf-8  w/ BOM decoded with utf-8     u'\ufeffABC'    # doesn't remove BOM if present.
utf-8  w/ BOM decoded with utf-8-sig u'ABC'          # removes BOM if present.
utf-16 w/ BOM decoded with utf-16    u'ABC'          # *requires* BOM to be present.
utf-16 w/ BOM decoded with utf-16le  u'\ufeffABC'    # doesn't remove BOM if present.

请注意,utf-16编解码器需要 BOM存在,否则Python将不知道数据是大端还是小端.

Answer 2

我在Python 3上遇到了这个问题并找到了这个问题(和解决方案).打开文件时,Python 3支持encoding关键字以自动处理编码.

没有它,BOM将包含在读取结果中:

>>> f = open('file', mode='r')
>>> f.read()
'\ufefftest'

给出正确的编码,结果中省略了BOM:

>>> f = open('file', mode='r', encoding='utf-8-sig')
>>> f.read()
'test'

只需2美分.

Answer 3

该字符是BOM或“字节顺序标记”。它通常作为文件的前几个字节接收，告诉您如何解释其余数据的编码。您只需删除该角色即可继续。不过，由于错误表明您正在尝试转换为“ascii”，因此您可能应该为您尝试执行的任何操作选择另一种编码。