Airflow：如何将 PDF 文件从 S3 存储桶下载到 Airflow

Question

尝试了下面的代码，但收到错误“无法找到凭据”def download()：

bucket = 'bucketname'
key = 'path and filename'
s3_resource = boto3.resource('s3')
my_bucket = s3_resource.Bucket(bucket)
objects = my_bucket.objects.filter(Prefix=key)
for obj = objects:
    path,filename = os.path.split(obj.key)
    my_bucket.download_file(obj.key, filename)

Answer 1

您需要 download_fileobj通过S3Hook.

我没有测试它，但它应该是这样的：

from tempfile import NamedTemporaryFile
from airflow.providers.amazon.aws.hooks.s3 import S3Hook
hook = S3Hook('my_aws_conn')
key_object = hook.get_key('your_path')
with NamedTemporaryFile("wb") as f:
     key_object.download_fileobj(Fileobj=f)
     f.flush()