如何从S3加载一个pickle文件以在AWS Lambda中使用？

Question

我当前正在尝试将S3中的腌制文件加载到AWS Lambda中并将其存储到列表中（腌制是列表）。

这是我的代码：

import pickle
import boto3

s3 = boto3.resource('s3')
with open('oldscreenurls.pkl', 'rb') as data:
    old_list = s3.Bucket("pythonpickles").download_fileobj("oldscreenurls.pkl", data)

即使文件存在，我也会收到以下错误：

FileNotFoundError: [Errno 2] No such file or directory: 'oldscreenurls.pkl'

有任何想法吗？

Answer 1

这是最简单的解决方案。您甚至可以使用S3FileSystem加载数据而无需在本地下载文件

from s3fs.core import S3FileSystem
s3_file = S3FileSystem()

data = pickle.load(s3_file.open('{}/{}'.format(bucket_name, file_path)))

Answer 2

如的文档所示download_fileobj，您需要以二进制写入模式打开文件并首先保存到文件。下载文件后，您可以将其打开以进行读取和修复。

import pickle
import boto3

s3 = boto3.resource('s3')
with open('oldscreenurls.pkl', 'wb') as data:
    s3.Bucket("pythonpickles").download_fileobj("oldscreenurls.pkl", data)

with open('oldscreenurls.pkl', 'rb') as data:
    old_list = pickle.load(data)

download_fileobj使用S3中的对象名称以及本地文件的句柄，然后将该对象的内容保存到文件中。此函数还有一个版本，该版本download_file采用文件名而不是打开的文件句柄并为您打开文件。

In this case it would probably be better to use S3Client.get_object though, to avoid having to write and then immediately read a file. You could also write to an in-memory BytesIO object, which acts like a file but doesn't actually touch a disk. That would look something like this:

import pickle
import boto3
from io import BytesIO

s3 = boto3.resource('s3')
with BytesIO() as data:
    s3.Bucket("pythonpickles").download_fileobj("oldscreenurls.pkl", data)
    data.seek(0)    # move back to the beginning after writing
    old_list = pickle.load(data)

Answer 3

超级简单的解决方案

import pickle
import boto3

s3 = boto3.resource('s3')
my_pickle = pickle.loads(s3.Bucket("bucket_name").Object("key_to_pickle.pickle").get()['Body'].read())