如何解码cp1252,它是十进制&#147而不是\ x93？

Question

我正在获取网页的源代码,编码是cp1252.Chrome正确显示页面.

这是我的代码:

import sys
from urllib.request import urlopen
from bs4 import BeautifulSoup, UnicodeDammit
import re
import codecs

url = "http://www.sec.gov/Archives/edgar/data/1400810/000119312513211026/d515005d10q.htm"
page = urlopen(url).read()
print(page)
# A little preview :
# b'...Regulation S-T (&#167;232.405 of this chapter) during the preceding 12 months (or for such shorter period that the\nregistrant was required to submit and post such files).&nbsp;&nbsp;&nbsp;&nbsp;Yes&nbsp;&nbsp;<FONT STYLE="FONT-FAMILY:WINGDINGS">&#120;</FONT>...'

soup = BeautifulSoup(page, from_encoding="cp1252")
print(str(soup).encode('utf-8'))
# Same preview section as above
# b'...Regulation S-T (\xc2\xa7232.405 of this chapter) during the preceding 12 months (or for such shorter period that the\nregistrant was required to submit and post such files).\xc2\xa0\xc2\xa0\xc2\xa0\xc2\xa0Yes\xc2\xa0\xc2\xa0<font style="FONT-FAMILY:WINGDINGS">x</font>'

从预览部分,我们可以看到
  \; =\xc2\xa0
§ =\xc2\xa7
x = x

对于cp1252编码标准,我指的是 http://en.wikipedia.org/wiki/Windows-1252#Code_page_layout 和/Lib/encodings/cp1252.py

当我使用BeautifulSoup(page,from_encoding ="cp1252")时,一些字符被正确编码,但有些字符则没有.

人物| 十进制编码| cp1252-> utf-8编码
"| “ |\xc2\x93(错误)
"| ” |\xc2\x94(错误)
X | x |\xc2\x92(错误)
§| § |\xc2\xa7(ok)
þ| þ
¨| ¨
'| ’ |\xc2\x92(错误)
- | –

我使用此代码来获得等价:

characters = "’ “ ” X § þ ¨ ' –"
list = characters.split()

for ch in list:
    print(ch)
    cp1252 = ch.encode('cp1252')
    print(cp1252)

    decimal = cp1252[0]

    special = "&#" + str(decimal)
    print(special)
    print(ch.encode('utf-8'))
    print()

offenders = [120, 146]

for n in offenders:
    toHex = hex(n)
    print(toHex)
print()

#120
off = b'\x78'
print(off)
buff = off.decode('cp1252')
print(buff)
uni = buff.encode('utf-8')
print(uni)
print()

#146
off = b'\x92'
print(off)
buff = off.decode('cp1252')
print(buff)
uni = buff.encode('utf-8')
print(uni)
print()

产量

’
b'\x92'
&#146
b'\xe2\x80\x99'

“
b'\x93'
&#147
b'\xe2\x80\x9c'

”
b'\x94'
&#148
b'\xe2\x80\x9d'

X
b'X'
&#88
b'X'

§
b'\xa7'
&#167
b'\xc2\xa7'

þ
b'\xfe'
&#254
b'\xc3\xbe'

¨
b'\xa8'
&#168
b'\xc2\xa8'

'
b"'"
&#39
b"'"

–
b'\x96'
&#150
b'\xe2\x80\x93'

0x78
0x92

b'x'
x
b'x'

b'\x92'
’
b'\xe2\x80\x99'

一些角色未能像编辑怪物那样复制粘贴到编辑器上并且很奇怪,所以我添加了一些代码来处理它.

对于"”",我可以对get\xe2\x80\x9d而不是\ xc2\x94做什么？

我的设置:
Windows 7
终端:chcp 1252 + Lucida Console字体
Python 3.3
BeautifulSoup 4

期待您的回答

Answer 1

这就是我最终使用的

def reformatCp1252(match):
    codePoint = int(match.group(1))

    if 128 <= codePoint <= 159:
        return bytes([codePoint])
    else:
        return match.group()

localPage = urlopen(r_url).read()
formatedPage = re.sub(b'&#(\d+);', reformatCp1252, localPage, flags=re.I)
localSoup = BeautifulSoup(formatedPage, "lxml", from_encoding="windows-1252")

注意：我在windows7中使用bs4和python3.3

我发现BeautifulSoup的from_encoding真的不重要，你可以输入utf-8或windows-1252，它给出了完整的utf-8编码替换windows-1252编码到 utf-8。
基本上所有的代码点都被解释为 utf-8 和单字节 \x? 被解释为 windows-1252。

据我所知，windows-1252 中只有 128 到 159 的字符与 utf-8 字符不同。

例如，混合编码（windows-1252：\x93 和 \x94 以及 utf-8：Ÿ）将仅输出 utf-8 格式的转换。

byteStream = b'\x93Hello\x94 (\xa7232.405 of this chapter) Ÿ \x87'
# with code above
print(localSoup.encode('utf-8'))
# and you can see that \x93 was transformed to its utf-8 equivalent.