Flask 大文件下载

Question

从 Flask 下载文件时发生内存错误。该文件的大小约为 100 兆字节。我该如何解决？

烧瓶下载代码

return send_from_directory(s_trash_path, s_zip_name, mimetype='zip', as_attachment=True)

错误代码

[2018-07-21 16:11:22,328] ERROR in app: Exception on /ec-fileupload/download/select [POST]
Traceback (most recent call last):
  File "/home/venv_ec_fileupload/lib/python3.6/site-packages/flask/app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/venv_ec_fileupload/lib/python3.6/site-packages/flask/app.py", line 1615, in full_dispatch_request
    return self.finalize_request(rv)
  File "/home/venv_ec_fileupload/lib/python3.6/site-packages/flask/app.py", line 1632, in finalize_request
    response = self.process_response(response)
  File "/home/venv_ec_fileupload/lib/python3.6/site-packages/flask/app.py", line 1856, in process_response
    response = handler(response)
  File "./app/__init__.py", line 170, in after_request
    s_data = resp.get_data()
  File "/home/venv_ec_fileupload/lib/python3.6/site-packages/werkzeug/wrappers.py", line 987, in get_data
    rv = b''.join(self.iter_encoded())
MemoryError

Answer 1

如果你提供二进制文件，你不应该遍历行，因为它基本上只包含一个“行”，这意味着你仍然一次将整个文件加载到 RAM 中。

读取大文件的唯一正确方法是通过块：

CHUNK_SIZE = 8192
def read_file_chunks(path):
    with open(path, 'rb') as fd:
        while 1:
            buf = fd.read(CHUNK_SIZE)
            if buf:
                yield buf
            else:
                break

然后调用stream_with_context这个块读取器是安全的，例如，如果您提供视频文件：

@app.route('/videos/<name>')
def serve_video(name):
    fp = resource_path_for(name)
    if fp.exists():
        return Response(
            stream_with_context(read_file_chunks(fp)),
            headers={
                'Content-Disposition': f'attachment; filename={name}'
            }
        )
    else:
        raise exc.NotFound()

在幕后，Flask 响应过程获取每个块（来自生成器read_file_chunks(fp)）并在加载下一个块之前将其刷新到连接。刷新后，块数据不再被引用并被垃圾收集器清理，因此不会有太多块同时留在RAM中。