小编Jus*_*ith的帖子

Python:将大型网页保存到文件

首先让我说,我不是编程新手,但对python来说还是新手.

我用urllib2编写了一个程序,它请求一个我想要保存到文件的网页.网页大约300KB,这并没有让我觉得特别大,但似乎足以给我带来麻烦,所以我称之为'大'.我正在使用一个简单的调用直接从返回的对象复制urlopen到文件中:

file.write(webpage.read())

但它只会坐几分钟,试图写入文件,我最终收到以下内容:

Traceback (most recent call last):
  File "program.py", line 51, in <module>
    main()
  File "program.py", line 43, in main
    f.write(webpage.read())
  File "/usr/lib/python2.7/socket.py", line 351, in read
    data = self._sock.recv(rbufsize)
  File "/usr/lib/python2.7/httplib.py", line 541, in read
    return self._read_chunked(amt)
  File "/usr/lib/python2.7/httplib.py", line 592, in _read_chunked
    value.append(self._safe_read(amt))
  File "/usr/lib/python2.7/httplib.py", line 649, in _safe_read
    raise IncompleteRead(''.join(s), amt)
httplib.IncompleteRead: IncompleteRead(6384 bytes read, 1808 more expected)

Run Code Online (Sandbox Code Playgroud)

我不知道为什么这会让节目如此悲痛？

编辑|

这是我如何检索页面

jar = cookielib.CookieJar()

cookie_processor = urllib2.HTTPCookieProcessor(jar);

opener = urllib2.build_opener(cookie_processor)
urllib2.install_opener(opener)

requ_login = …

Run Code Online (Sandbox Code Playgroud)

python file urllib2

Jus*_*ith

2011 11-23

5
推荐指数

1
解决办法

4338
查看次数

lxml.etree.XMLSyntaxError:htmlParseEntityRef:expecting';'

我试图找出python lxml api,但遇到了一个特殊的问题.我安装了以下库版本:

libxml2:2.7.8
libxslt:1.1.26

当我运行以下代码时:

html = open('file.html', 'r')
context = etree.iterparse(StringIO(html), events=("start", "end"), html='true')
for event, element in context:
    #do stuff

Run Code Online (Sandbox Code Playgroud)

编辑:

事实证明这是一个解析错误.我将html移动到一个文件(如下所示)

<html>
    <head></head>
    <body>
        <table>
            <tr>
                <td>image</td>
                <a href="relative.phtml?with=querystring&blah=blah">blah\n(blah)</a></td>
                <td>   35   </td>
                <td>   28   </td>
                <td><b>-7</b></td>
                <td>   
                23,000    </td>
                <td>   373,000   </td>
                <td>   644,000   </td>
                <td>+72.65%</td>
            </tr>
            <tr>
                <td>image</td>
                <td><a href="relative.phtml?with=querystring&blah=blah">blah\n(blah)</a></td>
                <td>   35   </td>
                <td>   28   </td>
                <td><b>-7</b></td>
                <td>   
                23,000    </td>
                <td>   373,000   </td>
                <td>   644,000   </td>
                <td>+72.65%</td>
            </tr>
        </table>
    </body>
</html>

Run Code Online (Sandbox Code Playgroud)

我现在收到这个错误: