从ASP.NET Core Web API将多GB文件流式传输到AWS S3

Question

我希望从ASP.NET Core Web API在AWS S3存储桶中创建一个大型(多GB)文件.该文件足够大,我希望Stream在将其上传到AWS S3之前不加载到内存中.

使用PutObjectAsync()我Stream之前必须预先填充它,然后将其传递给AWS SDK,如下图所示:

var putObjectRequest = new PutObjectRequest
{
    BucketName = "my-s3-bucket",
    Key = "my-file-name.txt",
    InputStream = stream
};
var putObjectResponse = await amazonS3Client.PutObjectAsync(putObjectRequest);

我理想的模式将涉及AWS SDK返回StreamWriter我可以Write()多次(各种)然后Finalise()当我完成时.

关于我的挑战的两个问题:

• 我是否误导了必须预先填充Stream之前的电话PutObjectAsync()？
• 我该如何上传我的大(多GB)文件？

Answer 1

对于这种情况，AWS 文档提供了两个选项：

• 使用 AWS .NET SDK 进行分段上传（高级 API）
• 使用 AWS .NET SDK 进行分段上传（低级 API）

高级API只建议你创建一个TransferUtilityUploadRequest带有PartSize指定，因此类本身可以上传文件，而无需自行维护的上传。在这种情况下，您可以通过订阅StreamTransferProgress事件获取分段上传的进度。您可以上传文件、流或目录。

显然，低级 API更复杂，但更灵活 - 您可以启动上传，然后在循环中上传文件的下一部分。文档中的示例代码：

var s3Client = new AmazonS3Client(Amazon.RegionEndpoint.USEast1);

// List to store upload part responses.
var uploadResponses = new List<UploadPartResponse>();

// 1. Initialize.
var initiateRequest = new InitiateMultipartUploadRequest
    {
        BucketName = existingBucketName,
        Key = keyName
    };

var initResponse = s3Client.InitiateMultipartUpload(initRequest);

// 2. Upload Parts.
var contentLength = new FileInfo(filePath).Length;
var partSize = 5242880; // 5 MB

try
{
    long filePosition = 0;
    for (var i = 1; filePosition < contentLength; ++i)
    {
        // Create request to upload a part.
        var uploadRequest = new UploadPartRequest
            {
                BucketName = existingBucketName,
                Key = keyName,
                UploadId = initResponse.UploadId,
                PartNumber = i,
                PartSize = partSize,
                FilePosition = filePosition,
                FilePath = filePath
            };

       // Upload part and add response to our list.
       uploadResponses.Add(s3Client.UploadPart(uploadRequest));

       filePosition += partSize;
   }

   // Step 3: complete.
   var completeRequest = new CompleteMultipartUploadRequest
       {
           BucketName = existingBucketName,
           Key = keyName,
           UploadId = initResponse.UploadId,
        };

   // add ETags for uploaded files
   completeRequest.AddPartETags(uploadResponses);

   var completeUploadResponse = s3Client.CompleteMultipartUpload(completeRequest);     
}
catch (Exception exception)
{
    Console.WriteLine("Exception occurred: {0}", exception.ToString());
    var abortMPURequest = new AbortMultipartUploadRequest
        {
            BucketName = existingBucketName,
            Key = keyName,
            UploadId = initResponse.UploadId
        };
    s3Client.AbortMultipartUpload(abortMPURequest);
}

的异步版本UploadPart也可用，因此如果您需要完全控制上传，您应该调查该路径。