如何使用boto3将S3对象保存到文件

Question

我正在尝试用AWS的新boto3客户端做一个"hello world" .

我使用的用例非常简单:从S3获取对象并将其保存到文件中.

在boto 2.XI会这样做:

import boto
key = boto.connect_s3().get_bucket('foo').get_key('foo')
key.get_contents_to_filename('/tmp/foo')

在博托3.我找不到干净的方法来做同样的事情,所以我手动迭代"Streaming"对象:

import boto3
key = boto3.resource('s3').Object('fooo', 'docker/my-image.tar.gz').get()
with open('/tmp/my-image.tar.gz', 'w') as f:
    chunk = key['Body'].read(1024*8)
    while chunk:
        f.write(chunk)
        chunk = key['Body'].read(1024*8)

要么

import boto3
key = boto3.resource('s3').Object('fooo', 'docker/my-image.tar.gz').get()
with open('/tmp/my-image.tar.gz', 'w') as f:
    for chunk in iter(lambda: key['Body'].read(4096), b''):
        f.write(chunk)

它工作正常.我想知道是否有任何"本机"boto3功能可以执行相同的任务？

Answer 1

最近进入Boto3的定制有助于此(除其他事项外).它目前在低级别S3客户端上公开,可以像这样使用:

s3_client = boto3.client('s3')
open('hello.txt').write('Hello, world!')

# Upload the file to S3
s3_client.upload_file('hello.txt', 'MyBucket', 'hello-remote.txt')

# Download the file from S3
s3_client.download_file('MyBucket', 'hello-remote.txt', 'hello2.txt')
print(open('hello2.txt').read())

这些函数将自动处理读/写文件以及为大文件并行执行分段上传.

Answer 2

boto3现在有一个比客户端更好的界面:

resource = boto3.resource('s3')
my_bucket = resource.Bucket('MyBucket')
my_bucket.download_file(key, local_filename)

这本身并不比client接受的答案好很多(尽管文档说它在重试失败时上传和下载方面做得更好)但考虑到资源通常更符合人体工程学(例如,s3 存储桶和对象资源)比客户端方法更好)这可以让你留在资源层而不必下拉.

Resources 通常可以以与客户端相同的方式创建,并且它们采用所有或大多数相同的参数,并将它们转发给内部客户端.

Answer 3

对于那些想要模拟set_contents_from_string类似boto2方法的人,你可以试试

import boto3
from cStringIO import StringIO

s3c = boto3.client('s3')
contents = 'My string to save to S3 object'
target_bucket = 'hello-world.by.vor'
target_file = 'data/hello.txt'
fake_handle = StringIO(contents)

# notice if you do fake_handle.read() it reads like a file handle
s3c.put_object(Bucket=target_bucket, Key=target_file, Body=fake_handle.read())

对于Python3:

在python3中,StringIO和cStringIO都消失了.使用StringIO导入如:

from io import StringIO

要支持这两个版本:

try:
   from StringIO import StringIO
except ImportError:
   from io import StringIO

Answer 4

# Preface: File is json with contents: {'name': 'Android', 'status': 'ERROR'}

import boto3
import io

s3 = boto3.resource('s3')

obj = s3.Object('my-bucket', 'key-to-file.json')
data = io.BytesIO()
obj.download_fileobj(data)

# object is now a bytes string, Converting it to a dict:
new_dict = json.loads(data.getvalue().decode("utf-8"))

print(new_dict['status']) 
# Should print "Error"

Answer 5

注意：我假设您已经单独配置了身份验证。下面的代码是从S3存储桶下载单个对象。

import boto3

#initiate s3 client 
s3 = boto3.resource('s3')

#Download object to the file    
s3.Bucket('mybucket').download_file('hello.txt', '/tmp/hello.txt')

Answer 6

如果您想下载文件的版本，则需要使用get_object.

import boto3

bucket = 'bucketName'
prefix = 'path/to/file/'
filename = 'fileName.ext'

s3c = boto3.client('s3')
s3r = boto3.resource('s3')

if __name__ == '__main__':
    for version in s3r.Bucket(bucket).object_versions.filter(Prefix=prefix + filename):
        file = version.get()
        version_id = file.get('VersionId')
        obj = s3c.get_object(
            Bucket=bucket,
            Key=prefix + filename,
            VersionId=version_id,
        )
        with open(f"{filename}.{version_id}", 'wb') as f:
            for chunk in obj['Body'].iter_chunks(chunk_size=4096):
                f.write(chunk)

参考： https: //botocore.amazonaws.com/v1/documentation/api/latest/reference/response.html