标签: boto3

在旧boto库是很简单的使用proxy,proxy_port,proxy_user和proxy_pass当你打开一个连接参数.但是,我找不到在boto3上以编程方式定义代理参数的任何等效方法.:(

我正在尝试从我在AWS中运行的Hadoop进程中读取一些日志.日志存储在S3文件夹中,并具有以下路径.

bucketname = name key = y/z/stderr.gz这里Y是集群ID,z是文件夹名称.这两者都充当AWS中的文件夹(对象).所以完整路径就像x/y/z/stderr.gz.

现在我想解压缩.gz文件并读取文件的内容.我不想将此文件下载到我的系统想要将内容保存在python变量中.

这是我到现在为止所尝试的.

bucket_name = "name"
key = "y/z/stderr.gz"
obj = s3.Object(bucket_name,key)
n = obj.get()['Body'].read()

这给了我一种不可读的格式.我也试过了

n = obj.get()['Body'].read().decode('utf-8')

这给出了错误utf8'编解码器无法解码位置1中的字节0x8b:无效的起始字节.

我也试过了

gzip = StringIO(obj)
gzipfile = gzip.GzipFile(fileobj=gzip)
content = gzipfile.read()

这将返回错误IOError:不是gzip压缩文件

不确定如何解码此.gz文件.

编辑 - 找到解决方案.需要传递n并使用BytesIO

gzip = BytesIO(n)

为什么python在安装pip或pip3显示它时没有找到boto3 .

我在Mac机器上.尝试/不使用pip/pip3 sudo.尝试更改PATH变量以及几条路径,但没有运气.

路径:/Library/Frameworks/Python.framework/Versions/3.5/bin:/Users/arun/.sdkman/candidates/gradle/current/bin:/Users/arun/aks/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin

和

$ python -c 'import sys; print sys.path'|tr ',' '\012'
[''
 '/Library/Python/2.7/site-packages/RBTools-0.5.7-py2.7.egg'
 '/Library/Python/2.7/site-packages/wfawsclitools-0.1-py2.7.egg'
 '/Library/Python/2.7/site-packages/PyYAML-3.12-py2.7-macosx-10.12-intel.egg'
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip'
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7'
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin'
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac'
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages'
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk'
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old'
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload'
 '/Library/Python/2.7/site-packages'
 '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python'
 '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC']

和

$ python -m site
sys.path = [
    '/Users/arun/ubuntu-vagrant-box',
    '/Library/Python/2.7/site-packages/RBTools-0.5.7-py2.7.egg',
    '/Library/Python/2.7/site-packages/wfawsclitools-0.1-py2.7.egg',
    '/Library/Python/2.7/site-packages/PyYAML-3.12-py2.7-macosx-10.12-intel.egg',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload',
    '/Library/Python/2.7/site-packages',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC',
]
USER_BASE: '/Users/arun/Library/Python/2.7' (doesn't exist)
USER_SITE: '/Users/arun/Library/Python/2.7/lib/python/site-packages' (doesn't exist)
ENABLE_USER_SITE: True

到目前为止我跑的步骤:

[arun@ip-10.11.33-65 ~/aks] $ …

在boto3或botocore中,我该如何设置相应的请求重试次数？

例如在boto2中

from boto import config
config.set('Boto', 'num_retries', '20')

我如何在boto3中执行此操作？我试过了

conn._session.set_config_variable("num_retries", "20")

但是当我然后get_config_variable("num_retries"),None返回.

我有一个AWS_ACCESS_KEY_ID和一个AWS_SECRET_KEY.这些是活动凭据,因此它们属于属于AWS账户的活动用户.如何使用Boto3找到此AWS账户的ID？

我一直在使用的方法是将Collection转换为List并查询长度:

s3 = boto3.resource('s3')
bucket = s3.Bucket('my_bucket')
size = len(list(bucket.objects.all()))

然而,这迫使整个系列的解决方案并且首先避免了使用Collection的好处.有一个更好的方法吗？

我在使用AWS Boto3使用推荐的KeyConditionExpression同时使用散列键和范围键查询DynamoDB时遇到问题.我附上了一个示例查询:

import boto3
from boto3 import dynamodb
from boto3.session import Session
dynamodb_session = Session(aws_access_key_id=AWS_KEY,
              aws_secret_access_key=AWS_PASS,
              region_name=DYNAMODB_REGION)
dynamodb = dynamodb_session.resource('dynamodb')
table=dynamodb.Table(TABLE_NAME)
request = {
    'ExpressionAttributeNames': {
        '#n0': 'hash_key',
        '#n1': 'range_key'
    },
    'ExpressionAttributeValues': {
        ':v0': {'S': MY_HASH_KEY},
        ':v1': {'N': GT_RANGE_KEY}
    },
    'KeyConditionExpression': '(#n0 = :v0) AND (#n1 > :v1)',
    'TableName': TABLE_NAME
}
response = table.query(**request)

当我使用以下方案对表执行此操作时:

Table Name: TABLE_NAME
Primary Hash Key: hash_key (String)
Primary Range Key: range_key (Number)

我收到以下错误,我不明白为什么:

ClientError: An error occurred (ValidationException) when calling the Query operation: Invalid …

我正在尝试使用boto3创建一个现场实例.虽然我遵循API文档,但我收到了一个我无法弄清楚的例外.我正在使用的代码是:

import boto3
import datetime
client = boto3.client('ec2')
response = client.request_spot_instances(
    DryRun=False,
    SpotPrice='0.10',
    ClientToken='string',
    InstanceCount=1,
    Type='one-time',
    LaunchSpecification={
        'ImageId': 'ami-fce3c696',
        'KeyName': 'awskey.pem',
        'SecurityGroups': ['sg-709f8709'],
        'InstanceType': 'm4.large',
        'Placement': {
            'AvailabilityZone': 'us-east-1a',
        },
        'BlockDeviceMappings': [
            {
                'Ebs': {
                    'SnapshotId': 'snap-f70deff0',
                    'VolumeSize': 100,
                    'DeleteOnTermination': True,
                    'VolumeType': 'gp2',
                    'Iops': 300,
                    'Encrypted': False
                },
            },
        ],

        'EbsOptimized': True,
        'Monitoring': {
            'Enabled': True
        },
        'SecurityGroupIds': [
            'sg-709f8709',
        ]
    }
)

我收到以下例外:

botocore.exceptions.ClientError: An error occurred (InvalidParameterValue) when calling the RequestSpotInstances operation: Value () for parameter …

随着AWS扩展并添加新区域,我希望我的代码能够自动检测到这一点.目前,"选择您的区域"是硬编码的,但我想解析以下只为RegionName.

import boto3

ec2 = boto3.client('ec2')
regions = ec2.describe_regions()
print(regions)

我的输出是JSON,如下所示:

{'Regions':[{'Endpoint':'ec2.ap-south-1.amazonaws.com','RegionName':'ap-south-1'},{'Endpoint':'ec2.eu-west- 1.amazonaws.com','RegionName':'eu-west-1'},{'Endpoint':'ec2.ap-southeast-1.amazonaws.com','RegionName':'ap-southeast-1' }]}

为了空间,我已经删除了重复数据和ResponseMetadata.

如何将RegionName解析为列表？

我的代码的一切工作.我目前面临的唯一陷阱是我无法在S3存储桶中指定我想要放入文件的文件夹.这就是我所拥有的:

s3.meta.client.upload_file('/tmp/'+filename, '<bucket-name>', filename)

我试过了两个:

s3.meta.client.upload_file('/tmp/'+filename, '<bucket-name>/folder/', filename)

和:

s3.meta.client.upload_file('/tmp/'+filename, '<bucket-name>', '/folder/'+filename)

如果有人有任何关于如何将其指向特定文件夹的提示(如果可能的话),请告诉我!