Python请求 - ChunkedEncodingError(e) - requests.iter_lines

Question

我正在ChunkedEncodingError(e)使用Python请求.我正在使用以下内容来删除JSON:

r = requests.get(url, headers=auth, stream=True)

并使用回车作为分隔符迭代每一行,这就是此API区分不同JSON事件的方式.

for d in r.iter_lines(delimiter="\n"):
    d += "\n"
    sock.send(d)

我正在对回车进行分界,然后将其添加回作为端点我正在推动日志实际上期望在每个事件结束时回车.这似乎适用于大约100k日志文件.当我尝试拨打更大的电话时,我会抛出以下内容:

for d in r.iter_lines(delimiter="\n"):
logs_1           |   File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 783, in iter_lines
logs_1           |     for chunk in self.iter_content(chunk_size=chunk_size, decode_unicode=decode_unicode):
logs_1           |   File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 742, in generate
logs_1           |     raise ChunkedEncodingError(e)
logs_1           | requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))

更新:我发现API NoneType也在某个时刻发回.那么如何在响应中的某处解释这个空字节而不会破坏一切？每个单独的事件都以a结束\n,我需要能够单独检查每个事件.我应该把内容分块而不是iter_lines？然后确保NoneType块中没有？这样我就不会试图iter_lines超过它NoneType并且它会爆炸？

Answer 1

ChunkedEncodingError 是由: httplib.IncompletedRead

import httplib

def patch_http_response_read(func):
    def inner(*args):
        try:
            return func(*args)
        except httplib.IncompleteRead, e:
            return e.partial
    return inner

httplib.HTTPResponse.read = patch_http_response_read(httplib.HTTPResponse.read)

我认为这可能是一个补丁.它允许您处理有缺陷的http服务器.

大多数服务器传输所有数据,但由于实施错误,它们会错误地关闭会话,httplib会引发错误并掩盖您宝贵的字节.