Python请求-尽管使用了流媒体上传，但仍出现MemoryError

Question

根据文档，应该通过给Request一个类似文件的对象而不是文件的内容来进行不占用大量内存的上载。好的，所以我在代码中这样做：

files = {'md5': ('', md5hash),
         'modified': ('', now),
         'created': ('', now),
         'file': (os.path.basename(url), fileobject, 'application/octet-stream', {'Content-Transfer-Encoding':'binary'})}
r = s.post(url, data=content, params=params, files=files, headers=headers)

看着它在我的计算机上运行时（带有2.8 GB的文件），它开始以惊人的速度消耗掉内存，直到它使用的内存达到约89％时才恢复工作。然后，它失败并显示以下输出：

  File "***.py", line 644, in post
    r = s.post(url, data=content, params=params, files=files, headers=headers)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 424, in post
    return self.request('POST', url, data=data, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests_cache/core.py", line 110, in request
    hooks, stream, verify, cert)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 348, in request
    prep = self.prepare_request(req)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 286, in prepare_request
    hooks=merge_hooks(request.hooks, self.hooks),
  File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 289, in prepare
    self.prepare_body(data, files)
  File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 426, in prepare_body
    (body, content_type) = self._encode_files(files, data)
  File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 144, in _encode_files
    body, content_type = encode_multipart_formdata(new_fields)
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/filepost.py", line 101, in encode_multipart_formdata
    return body.getvalue(), content_type
MemoryError

对于较小的文件，它可以正常工作，但这样做仍然会占用大量内存。我误会了吗？

编辑：

看到Martijn Pieters的回答后，我将代码更改为：

    files = {'md5': ('', md5hash),
             'modified': ('', now),
             'created': ('', now),
             'file': (os.path.basename(url), fileobject, 'application/octet-stream')}
    m = requests_toolbelt.MultipartEncoder(fields=files)
    headers['content-type'] = m.content_type
    r = s.post(url, data=m, params=params, headers=headers)

我必须删除{'Content-Transfer-Encoding':'binary'}，因为它似乎不受支持，并导致出现此错误消息：

  File "***.py", line 647, in post
    m = requests_toolbelt.MultipartEncoder(fields=files)
  File "/usr/local/lib/python2.7/dist-packages/requests_toolbelt/multipart/encoder.py", line 89, in __init__
    self._prepare_parts()
  File "/usr/local/lib/python2.7/dist-packages/requests_toolbelt/multipart/encoder.py", line 171, in _prepare_parts
    self.parts = [Part.from_field(f, enc) for f in fields]
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/filepost.py", line 44, in iter_field_objects
    yield RequestField.from_tuples(*field)
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/fields.py", line 97, in from_tuples
filename, data = value
ValueError: too many values to unpack

（在使用多部分编码器时，是否仍可以设置此标头？我更希望它在那里。）

但是，即使删除了该标头，它仍然无法工作，因为现在我收到了以下错误消息：

  File "***.py", line 647, in post
    r = s.post(url, data=m, params=params, headers=headers)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 424, in post
    return self.request('POST', url, data=data, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests_cache/core.py", line 114, in request
    main_key = self.cache.create_key(response.request)
  File "/usr/local/lib/python2.7/dist-packages/requests_cache/backends/base.py", line 156, in create_key
    key.update(_to_bytes(request.body))
TypeError: must be convertible to a buffer, not MultipartEncoder

有任何想法吗？我承认我对此很陌生，并且这些错误消息（在编程中经常如此）没有帮助。

Answer 1

您没有流式上载，因为requests仅当整个主体都来自打开的文件对象时，才可以这样做。它将仍然将所有文件读入内存以构建一个多部分的帖子正文。

对于分段上传，请使用请求工具栏；它包括一个流式多部分数据编码器：

from requests_toolbelt import MultipartEncoder
import requests

files = {
    'md5': ('', md5hash),
    'modified': ('', now),
    'created': ('', now),
    'file': (os.path.basename(url), fileobject, 'application/octet-stream')
}
m = MultipartEncoder(fields=dict(files, **params))
headers['content-type'] = m.content_type

r = s.post(url, data=m, headers=headers)
r = requests.post('http://httpbin.org/post', data=m, headers=headers)

的第一个参数MultipartEncoder进行解析与所述iter_field_objects()功能从所述urllib3文库; 这意味着它可以是键-值对的字典，或序列（列表，元组）RequestField()的对象。

当像我在上面那样传递字典时，每个键值对都使用进行解析RequestField.from_tuples()，并且您只能指定字段名称，值以及文件名和mimetype（可选）。不支持额外的标题。我在上面的示例中使用了该选项。

如果要将Content-Transfer-Encoding标题添加到该file字段，则需要使用一系列RequestField对象：

from requests.packages.urllib3.fields import RequestField

fields = [RequestField.from_tuples(*p) for p in params.iteritems()]
fields.extend([
    RequestField('md5', md5hash),
    RequestField('modified', now),
    RequestField('created', now),
    RequestField(
        'file', fileobject, 'application/octet-stream',
        {'Content-Transfer-Encoding':'binary'}),
])

注意，您不能将流请求与请求缓存项目结合在一起；后者需要访问请求的整个内容以生成缓存键。

您必须修补该requests_cache.backends.base.BaseCache.create_key方法以处理MultipartEncoder对象，并为主体提供某种哈希键。但是，这超出了此问题的范围。