python-请求在下载大量文件时挂起

Question

我正在尝试使用 python-request 包从网络下载大量文件（例如 10k+），每个文件大小从几 k 到最大 100mb。

我的脚本可以正常运行大约 3000 个文件，但突然它会挂起。我按住 ctrl-c 并看到它卡在

    r = requests.get(url, headers=headers, stream=True)
  File "/Library/Python/2.7/site-packages/requests/api.py", line 55, in get
    return request('get', url, **kwargs)
  File "/Library/Python/2.7/site-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Library/Python/2.7/site-packages/requests/sessions.py", line 456, in request
    resp = self.send(prep, **send_kwargs)
  File "/Library/Python/2.7/site-packages/requests/sessions.py", line 559, in send
    r = adapter.send(request, **kwargs)
  File "/Library/Python/2.7/site-packages/requests/adapters.py", line 327, in send
    timeout=timeout
  File "/Library/Python/2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 493, in urlopen
    body=body, headers=headers)
  File "/Library/Python/2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 319, in _make_request
    httplib_response = conn.getresponse(buffering=True)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1045, in getresponse
    response.begin()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 409, in begin
    version, status, reason = self._read_status()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 365, in _read_status
    line = self.fp.readline(_MAXLINE + 1)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 476, in readline
    data = self._sock.recv(self._rbufsize)

这是我用于下载的 python 代码

basedir = os.path.dirname(filepath)
if not os.path.exists(basedir):
    os.makedirs(basedir)
r = requests.get(url, headers=headers, stream=True)
with open(filepath, 'w') as f:
    for chunk in r.iter_content(1024):
        if chunk:
            f.write(chunk)
            f.flush()

我不确定出了什么问题，如果有人有线索，请分享一些见解。谢谢。

Answer 1

这不是@alfasin 在他们的评论中链接的问题的重复。从您发布的（有限的）回溯来看，请求本身已挂起（第一行显示它正在执行r = requests.get(url, headers=headers, stream=True)）。

您应该做的是设置超时并捕获请求超时时引发的异常。获得 URL 后，在浏览器中或使用curl 尝试它以确保其正确响应，否则将其从要请求的 URL 列表中删除。如果您发现行为不当的 URL，请用它更新您的问题。