检查S3上的文件大小而不下载？

Question

我将客户文件上传到Amazon S3,我想添加一项功能来计算每个客户的文件大小.有没有办法"偷看"文件大小而不下载它们？我知道您可以从亚马逊控制面板查看,但我需要以编程方式进行操作.

Answer 1

向对象发送HTTP HEAD请求.HEAD请求将检索与GET请求相同的HTTP标头,但它不会检索对象的主体(节省带宽).然后,您可以从HTTP响应标头中解析出Content-Length标头值.

Answer 2

您可以简单地使用以下s3 ls命令：

aws s3 ls s3://mybucket --recursive --human-readable --summarize

输出

2013-09-02 21:37:53   10 Bytes a.txt
2013-09-02 21:37:53  2.9 MiB foo.zip
2013-09-02 21:32:57   23 Bytes foo/bar/.baz/a
2013-09-02 21:32:58   41 Bytes foo/bar/.baz/b
2013-09-02 21:32:57  281 Bytes foo/bar/.baz/c
2013-09-02 21:32:57   73 Bytes foo/bar/.baz/d
2013-09-02 21:32:57  452 Bytes foo/bar/.baz/e
2013-09-02 21:32:57  896 Bytes foo/bar/.baz/hooks/bar
2013-09-02 21:32:57  189 Bytes foo/bar/.baz/hooks/foo
2013-09-02 21:32:57  398 Bytes z.txt

Total Objects: 10
   Total Size: 2.9 MiB

参考：https : //docs.aws.amazon.com/cli/latest/reference/s3/ls.html

Answer 3

Node.js一:

const AWS = require('aws-sdk');
const s3 = new AWS.S3();

function sizeOf(key, bucket) {
    return s3.headObject({ Key: key, Bucket: bucket })
        .promise()
        .then(res => res.ContentLength);
}


// A test
sizeOf('ahihi.mp4', 'output').then(size => console.log(size));

Doc就在这里.

Answer 4

这是适用于使用 Java 和 Amazon 提供的 S3 java 库的任何人的解决方案。如果您正在使用，com.amazonaws.services.s3.AmazonS3您可以使用GetObjectMetadataRequest允许您查询对象长度的请求。

您必须使用的库是：

<!-- https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-s3 -->
<dependency>
    <groupId>com.amazonaws</groupId>
    <artifactId>aws-java-sdk-s3</artifactId>
    <version>1.11.511</version>
</dependency>

进口：

import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.services.s3.model.*;

以及获取内容长度所需的代码：

GetObjectMetadataRequest metadataRequest = new GetObjectMetadataRequest(bucketName, fileName);
final ObjectMetadata objectMetadata = s3Client.getObjectMetadata(metadataRequest);
long contentLength = objectMetadata.getContentLength();

在执行上述代码之前，您需要构建 S3 客户端。这是一些示例代码：

AWSCredentials credentials = new BasicAWSCredentials(
            accessKey,
            secretKey
);
s3Client = AmazonS3ClientBuilder.standard()
            .withRegion(clientRegion)
            .withCredentials(new AWSStaticCredentialsProvider(credentials))
            .build();

Answer 5

使用迈克尔的建议,我的成功代码如下所示:

require 'net/http'
require 'uri'

file_url = MyObject.first.file.url

url = URI.parse(file_url)
req = Net::HTTP::Head.new url.path
res = Net::HTTP.start(url.host, url.port) {|http|
  http.request(req)
}

file_length = res["content-length"]

Answer 6

我在 Python 中执行类似的操作来获取给定前缀下所有文件的累积大小：

import boto3

bucket = 'your-bucket-name'
prefix = 'some/s3/prefix/'

s3 = boto3.client('s3')

size = 0

result = s3.list_objects_v2(Bucket=bucket, Prefix=prefix)
size += sum([x['Size'] for x in result['Contents']])

while result['IsTruncated']:
    result = s3.list_objects_v2(
        Bucket=bucket, Prefix=prefix,
        ContinuationToken=result['NextContinuationToken'])
    size += sum([x['Size'] for x in result['Contents']])

print('Total size in MB: ' + str(size / (1000**2)))

Answer 7

.NET AWS SDK ---- ListObjectsRequest、ListObjectsResponse、S3Object

AmazonS3Client s3 = new AmazonS3Client();
SpaceUsed(s3, "putBucketNameHere");

static void SpaceUsed(AmazonS3Client s3Client, string bucketName)
    {
        ListObjectsRequest request = new ListObjectsRequest();
        request.BucketName = bucketName;
        ListObjectsResponse response = s3Client.ListObjects(request);
        long totalSize = 0;
        foreach (S3Object o in response.S3Objects)
        {
            totalSize += o.Size;
        }
        Console.WriteLine("Total Size of bucket " + bucketName + " is " +
            Math.Round(totalSize / 1024.0 / 1024.0, 2) + " MB");
    }

Answer 8

如果您希望对单个文件执行此操作，则可以aws s3api head-object仅使用它来获取元数据，而无需下载文件本身：

$ aws s3api head-object --bucket mybucket --key path/to/myfile.csv --query "ContentLength"

解释

• s3api head-object以 json 格式检索对象元数据
• --query "ContentLength"过滤 json 响应以获取正文的大小（以字节为单位）