如何在 S3 中列出前缀下的所有对象

Question

前缀下有超过 3k 个对象。我使用以下代码列出所有对象以获取其名称，但 API 仅检索 1000 个对象。s3_client = boto3.client('s3')

response = s3_client.list_objects(
    Bucket = "my-bucket",
    Prefix = "my-prefix",
    MaxKeys=50000
)


s3 = boto3.resource('s3')
bucket = s3.Bucket(S3)

print(len(response['Contents'])) # only retrieve 1000

Answer 1

使用分页器循环浏览多个页面。请参阅：创建分页器

import boto3

client = boto3.client('s3')
paginator = client.get_paginator('list_objects')
operation_parameters = {'Bucket': 'my-bucket',
                        'Prefix': 'my-prefix'}
page_iterator = paginator.paginate(**operation_parameters)
for page in page_iterator:
    print(page['Contents'])