使用boto3连接到S3加速端点

Question

我想从已激活加速的S3存储桶将文件下载到Python文件对象中.我遇到了一些资源,建议是否覆盖endpoint_url"s3-accelerate.amazonaws.com"和/或使用该use_accelerate_endpoint属性.

我已经尝试了两种,并且有几种变化,但每次都返回相同的错误.我尝试过的一个脚本是:

    from botocore.config import Config
    import boto3
    from io import BytesIO
    session = boto3.session.Session()
    s3 = session.client(                                                                            
           service_name='s3',                                                                  
           aws_access_key_id=<MY_KEY_ID>,               
           aws_secret_access_key=<MY_KEY>,
           region_name="us-west-2",
       config=Config(s3={"use_accelerate_endpoint": True,
                         "addressing_style": "path"}))
   input = BytesIO() 
   s3.download_fileobj(<MY_BUCKET>,<MY_KEY>, input)

返回以下错误:

            ---------------------------------------------------------------------------
            ClientError                               Traceback (most recent call last)
            <ipython-input-61-92b89b45f215> in <module>()
                 11      "addressing_style": "path"}))
                 12 input = BytesIO()
            ---> 13 s3.download_fileobj(bucket, filename, input)
                 14 
                 15 

            ~/Project/venv/lib/python3.5/site-packages/boto3/s3/inject.py in download_fileobj(self, Bucket, Key, Fileobj, ExtraArgs, Callback, Config)
                568             bucket=Bucket, key=Key, fileobj=Fileobj,
                569             extra_args=ExtraArgs, subscribers=subscribers)
            --> 570         return future.result()
                571 
                572 

            ~/Project//venv/lib/python3.5/site-packages/s3transfer/futures.py in result(self)
                 71             # however if a KeyboardInterrupt is raised we want want to exit
                 72             # out of this and propogate the exception.
            ---> 73             return self._coordinator.result()
                 74         except KeyboardInterrupt as e:
                 75             self.cancel()

            ~/Project/venv/lib/python3.5/site-packages/s3transfer/futures.py in result(self)
                231         # final result.
                232         if self._exception:
            --> 233             raise self._exception
                234         return self._result
                235 

            ~/Project/venv/lib/python3.5/site-packages/s3transfer/tasks.py in _main(self, transfer_future, **kwargs)
                253             # Call the submit method to start submitting tasks to execute the
                254             # transfer.
            --> 255             self._submit(transfer_future=transfer_future, **kwargs)
                256         except BaseException as e:
                257             # If there was an exception raised during the submission of task

            ~/Project/venv/lib/python3.5/site-packages/s3transfer/download.py in _submit(self, client, config, osutil, request_executor, io_executor, transfer_future)
                347                 Bucket=transfer_future.meta.call_args.bucket,
                348                 Key=transfer_future.meta.call_args.key,
            --> 349                 **transfer_future.meta.call_args.extra_args
                350             )
                351             transfer_future.meta.provide_transfer_size(

            ~/Project/venv/lib/python3.5/site-packages/botocore/client.py in _api_call(self, *args, **kwargs)
                310                     "%s() only accepts keyword arguments." % py_operation_name)
                311             # The "self" in this scope is referring to the BaseClient.
            --> 312             return self._make_api_call(operation_name, kwargs)
                313 
                314         _api_call.__name__ = str(py_operation_name)

            ~/Project/venv/lib/python3.5/site-packages/botocore/client.py in _make_api_call(self, operation_name, api_params)
                603             error_code = parsed_response.get("Error", {}).get("Code")
                604             error_class = self.exceptions.from_code(error_code)
            --> 605             raise error_class(parsed_response, operation_name)
                606         else:
                607             return parsed_response

            ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden

当我运行相同的脚本时"use_accelerate_endpoint": False它工作正常.但是,它在以下情况下返回相同的错误:

• 我endpoint_url用"s3-accelerate.amazonaws.com" 覆盖了
• 我定义 "addressing_style": "virtual"

跑步时 s3.get_bucket_accelerate_configuration(Bucket=<MY_BUCKET>)

我{..., 'Status': 'Enabled'}按预期得到了.

知道该代码有什么问题,我应该改变什么来正确查询该存储桶的加速端点？

在Ubuntu 17.04上使用python3.5和boto3 == 1.4.7,botocore == 1.7.43.

编辑:我也尝试过类似的上传脚本:

    from botocore.config import Config
    import boto3
    from io import BytesIO
    session = boto3.session.Session()
    s3 = session.client(                                                                            
           service_name='s3',                                                                  
           aws_access_key_id=<MY_KEY_ID>,               
           aws_secret_access_key=<MY_KEY>,
           region_name="us-west-2",
       config=Config(s3={"use_accelerate_endpoint": True,
                         "addressing_style": "virtual"}))
   output = BytesIO()
   output.seek(0)
   s3.upload_fileobj(output, <MY_BUCKET>,<MY_KEY>)

哪个没有use_accelerate_endpoint选项(所以我的键很好),但在True时返回此错误:

   ClientError: An error occurred (SignatureDoesNotMatch) when calling the PutObject operation: The request signature we calculated does not match the signature you provided. Check your key and signing method.

我也尝试了这两个addressing_style选项(虚拟和路径)

Answer 1

使用boto3==1.4.7和botocore==1.7.43。这是从启用了传输加速的存储桶中检索对象的一种方法。

import boto3
from botocore.config import Config
from io import BytesIO

config = Config(s3={"use_accelerate_endpoint": True})
s3_resource = boto3.resource("s3",
                             aws_access_key_id=<MY_KEY_ID>,               
                             aws_secret_access_key=<MY_KEY>,
                             region_name="us-west-2",
                             config=config)

s3_client = s3_resource.meta.client

file_object = BytesIO()
s3_client.download_fileobj(<MY_BUCKET>, <MY_KEY>, file_object)

请注意，客户端HEAD在GET.

其规范请求类似于以下内容：

CanonicalRequest:
HEAD
/<MY_KEY>

host:<MY_BUCKET>.s3-accelerate.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20200520T204128Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

HEAD请求失败的一些原因包括：

• 具有给定键的对象不存在或启用了严格的访问控制
• 无效证件
• 传输加速未启用