如何在Amazon S3存储桶中列出_all_对象？

Question

S3Client.ListObjects仅返回1000个对象.如何使用Amazon C#库检索所有现有对象的列表？

Answer 1

如前所述,Amazon S3确实需要使用AWS SDK for .NET列出密钥:

由于存储桶可以包含几乎无限数量的密钥,因此列表查询的完整结果可能非常大.为了管理大型结果集,Amazon S3使用分页将它们拆分为多个响应.每个列表键响应返回一个最多1,000个键的页面,其中有一个指示符,指示响应是否被截断.您发送一系列列表键请求,直到您收到所有键.

提到的指标是ObjectsResponse类的NextMarker属性- 它的用法在使用AWS SDK for .NET的完整示例列表键中说明,相关片段为:

static AmazonS3 client;
client = Amazon.AWSClientFactory.CreateAmazonS3Client(
                    accessKeyID, secretAccessKeyID);

ListObjectsRequest request = new ListObjectsRequest();
request.BucketName = bucketName;
do
{
   ListObjectsResponse response = client.ListObjects(request);

   // Process response.
   // ...

   // If response is truncated, set the marker to get the next 
   // set of keys.
   if (response.IsTruncated)
   {
        request.Marker = response.NextMarker;
   }
   else
   {
        request = null;
   }
} while (request != null);

Answer 2

请注意,上面的答案并未使用推荐的API来列出对象:http://docs.aws.amazon.com/AmazonS3/latest/API/v2-RESTBucketGET.html

以下代码段显示了新API的外观:

using (var s3Client = new AmazonS3Client(Amazon.RegionEndpoint.USEast1))
{
    ListObjectsV2Request request = new ListObjectsV2Request
    {
          BucketName = bucketName,
          MaxKeys = 10
    };
    ListObjectsV2Response response;
    do
    {
         response = await s3Client.ListObjectsV2Async(request);

         // Process response.
         // ...

         request.ContinuationToken = response.NextContinuationToken;

    } while (response.IsTruncated);        
}