如何将客户端上传可查看文件到Amazon S3？

Question

首先我要说的是,我通常非常不愿意发布这些问题,因为我总觉得有一个答案可以解决互联网上的所有问题.在花了无数个小时寻找这个问题的答案之后,我终于放弃了这个陈述.

假设

这有效:

s3.getSignedUrl('putObject', params);

我想做什么？

1. 使用getSignedUrl方法通过PUT(从客户端)将文件上载到Amazon S3
2. 允许任何人查看上传到S3的文件

注意:如果有一种更简单的方法允许客户端(iPhone)使用预先签名的URL上传到Amazon S3(并且没有公开客户端的凭据),我会全神贯注.

主要问题*

1. 查看AWS管理控制台时,上载的文件设置了空白权限和元数据.
2. 查看上载的文件时(即双击AWS管理控制台中的文件),我收到AccessDenied错误消息.

我试过了什么？

尝试#1:我的原始代码

在NodeJS中,我生成一个预先签名的URL,如下所示:

var params = {Bucket: mybucket, Key: "test.jpg", Expires: 600};
s3.getSignedUrl('putObject', params, function (err, url){
  console.log(url); // this is the pre-signed URL
});

预签名的URL看起来像这样:

https://mybucket.s3.amazonaws.com/test.jpg?AWSAccessKeyId=AABFBIAWAEAUKAYGAFAA&Expires=1391069292&Signature=u%2BrqUtt3t6BfKHAlbXcZcTJIOWQ%3D

现在我通过PUT上传文件

curl -v -T myimage.jpg https://mybucket.s3.amazonaws.com/test.jpg?AWSAccessKeyId=AABFBIAWAEAUKAYGAFAA&Expires=1391069292&Signature=u%2BrqUtt3t6BfKHAlbXcZcTJIOWQ%3D

问题
我得到上面列出的*主要问题

尝试#2:在PUT上添加内容类型和ACL

我也尝试在代码中添加Content-Type和x-amz-acl,方法是替换params,如下所示:

var params = {Bucket: mybucket, Key: "test.jpg", Expires: 600, ACL: "public-read-write", ContentType: "image/jpeg"};

然后我尝试了一个很好的'PUT:

curl -v -H "image/jpeg" -T myimage.jpg https://mybucket.s3.amazonaws.com/test.jpg?AWSAccessKeyId=AABFBIAWAEAUKAYGAFAA&Content-Type=image%2Fjpeg&Expires=1391068501&Signature=0yF%2BmzDhyU3g2hr%2BfIcVSnE22rY%3D&x-amz-acl=public-read-write

问题
我的终端输出一些错误:

-bash: Content-Type=image%2Fjpeg: command not found
-bash: x-amz-acl=public-read-write: command not found

我也得到了上面列出的*主要问题.

尝试#3:修改存储桶权限是公开的

下面列出的所有项目都在AWS管理控制台中勾选)

Grantee: Everyone can [List, Upload/Delete, View Permissions, Edit Permissions]
Grantee: Authenticated Users can [List, Upload/Delete, View Permissions, Edit Permissions]

存储策略

{
"Version": "2012-10-17",
"Statement": [
    {
        "Sid": "Stmt1390381397000",
        "Effect": "Allow",
        "Principal": {
            "AWS": "*"
        },
        "Action": "s3:*",
        "Resource": "arn:aws:s3:::mybucket/*"
    }
]
}

尝试#4:设置IAM权限

我将用户策略设置为:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:*",
      "Resource": "*"
    }
  ]
}

AuthenticatedUsers组策略是这样的:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1391063032000",
      "Effect": "Allow",
      "Action": [
        "s3:*"
      ],
      "Resource": [
        "*"
      ]
    }
  ]
}

尝试#5:设置CORS策略

我将CORS策略设置为:

<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
    <CORSRule>
        <AllowedOrigin>*</AllowedOrigin>
        <AllowedMethod>PUT</AllowedMethod>
        <AllowedMethod>POST</AllowedMethod>
        <AllowedMethod>DELETE</AllowedMethod>
        <AllowedMethod>GET</AllowedMethod>
        <MaxAgeSeconds>3000</MaxAgeSeconds>
        <AllowedHeader>*</AllowedHeader>
    </CORSRule>
</CORSConfiguration>

而且......现在我在这里.

Answer 1

更新

我有坏消息.根据SDK 2.1.6的发行说明http://aws.amazon.com/releasenotes/1473534964062833:

"The SDK will now throw an error if ContentLength is passed into an 
Amazon S3 presigned URL (AWS.S3.getSignedUrl()). Passing a 
ContentLength is not supported by the SDK, since it is not enforced on 
S3's side given the way the SDK is currently generating these URLs. 
See GitHub issue #457."

我发现在某些场合,必须包含ContentLength(特别是如果你的客户端传递它以便签名匹配),那么在其他场合,如果你在ContentLength中包含参数错误,getSignedUrl会抱怨:"预签名网址不支持contentlength ".我注意到当我更改正在进行呼叫的机器时,行为会发生变化.据推测,另一台机器与服务器场中的另一台亚马逊服务器建立了连接.

我只能猜测为什么行为在某些情况下存在,而在其他情况下不存在.也许亚马逊的所有服务器都没有完全升级？在任何一种情况下,为了处理这个问题,我现在尝试使用ContentLength,如果它给我参数错误,那么我在没有它的情况下再次调用getSignedUrl.这是解决SDK的这种奇怪行为的解决方法.

一个小例子......看起来不是很漂亮,但你明白了:

MediaBucketManager.getPutSignedUrl = function ( params, next ) {
    var _self = this;
    _self._s3.getSignedUrl('putObject', params, function ( error, data ) {
        if (error) {
            console.log("An error occurred retrieving a signed url for putObject", error);
            // TODO: build contextual error
            if (error.code == "UnexpectedParameter" && error.message.search("ContentLength") > -1) {
                if (params.ContentLength) delete params.ContentLength
                MediaBucketManager.getPutSignedUrl(bucket, key, expires, params, function ( error, data ) {
                    if (error) {
                        console.log("An error occurred retrieving a signed url for putObject", error);
                    } else {
                        console.log("Retrieved a signed url for putObject:", data);
                        return next(null, data)
                    }
                }); 
            } else {
                return next(error); 
            }
        } else {
            console.log("Retrieved a signed url for putObject:", data);
            return next(null, data);
        }
    });
};

所以,下面的内容并不完全正确(在某些情况下会更正,但在其他情况下会给你参数错误),但可能会帮助你入门.

老答案

看来(对于一个signedUrl将文件PUT输出到只存在公共读取ACL的S3),当向PUT发出请求时,会有几个标头被比较.将它们与传递给getSignedUrl的内容进行比较:

CacheControl: 'STRING_VALUE',
ContentDisposition: 'STRING_VALUE',
ContentEncoding: 'STRING_VALUE',
ContentLanguage: 'STRING_VALUE',
ContentLength: 0,
ContentMD5: 'STRING_VALUE',
ContentType: 'STRING_VALUE',
Expires: new Date || 'Wed De...'

请参阅此处的完整列表:http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#putObject-property

当你调用getSignedUrl时,你将传递一个'params'对象(在文档中相当清楚),包括Bucket,Key和Expires数据.这是一个(NodeJS)示例:

var params = { Bucket:bucket, Key:key, Expires:expires };
s3.getSignedUrl('putObject', params, function ( error, data ) {
    if (error) {
        // handle error
    } else {
        // handle data
    }
});

不太明确的是将ACL设置为'public-read':

var params = { Bucket:bucket, Key:key, Expires:expires, ACL:'public-read' };

传递标题的概念非常模糊,您期望客户端使用签名的URL将PUT操作传递给S3:

var params = {
    Bucket:bucket,
    Key:key,
    Expires:expires,
    ACL:'public-read',
    ContentType:'image/png',
    ContentLength:7469
};

在上面的示例中,我已经包含了ContentType和ContentLength,因为在javascript中使用XmlHTTPRequest时会包含这两个标头,并且在Content-Length的情况下无法更改.我怀疑像Curl这样的HTTP请求的其他实现会出现这种情况,因为它们在提交包含正文(数据)的HTTP请求时是必需的标头.

如果客户端在请求signedUrl时没有包含有关文件的ContentType和ContentLength数据,那么当需要将文件PUT到S3(使用该signedUrl)时,S3服务将找到客户端请求中包含的头文件(因为它们是必需的标题)但签名不会包含它们 - 因此,它们将不匹配,操作将失败.

因此,在进行getSignedUrl调用之前,您似乎必须知道文件的内容类型和内容长度为PUT到S3.这对我来说不是问题,因为我暴露了一个REST端点,允许我们的客户端在对PUT进行S3操作之前请求签名的URL.由于客户端可以访问要提交的文件(当时它们已准备好提交),因此客户端访问文件大小和类型并从我的端点请求带有该数据的签名URL是一项微不足道的操作.

Answer 2

根据@Reinsbrain请求,这是实现客户端上传到具有"公共读取"权限的服务器的Node.js版本.

BACKEND(NODE.JS)

var AWS = require('aws-sdk');
var AWS_ACCESS_KEY_ID = process.env.S3_ACCESS_KEY;
var AWS_SECRET_ACCESS_KEY = process.env.S3_SECRET;
AWS.config.update({accessKeyId: AWS_ACCESS_KEY_ID, secretAccessKey: AWS_SECRET_ACCESS_KEY});
var s3 = new AWS.S3();
var moment = require('moment');
var S3_BUCKET = process.env.S3_BUCKET;
var crypto = require('crypto');
var POLICY_EXPIRATION_TIME = 10;// change to 10 minute expiry time
var S3_DOMAIN = process.env.S3_DOMAIN;

exports.writePolicy = function (filePath, contentType, maxSize, redirect, callback) {
  var readType = "public-read";

  var expiration = moment().add('m', POLICY_EXPIRATION_TIME);//OPTIONAL: only if you don't want a 15 minute expiry

  var s3Policy = {
    "expiration": expiration,
    "conditions": [
      ["starts-with", "$key", filePath],
      {"bucket": S3_BUCKET},
      {"acl": readType},
      ["content-length-range", 2048, maxSize], //min 2kB to maxSize
      {"redirect": redirect},
      ["starts-with", "$Content-Type", contentType]
    ]
  };

  // stringify and encode the policy
  var stringPolicy = JSON.stringify(s3Policy);
  var base64Policy = Buffer(stringPolicy, "utf-8").toString("base64");

  // sign the base64 encoded policy
  var testbuffer = new Buffer(base64Policy, "utf-8");

  var signature = crypto.createHmac("sha1", AWS_SECRET_ACCESS_KEY)
    .update(testbuffer).digest("base64");

  // build the results object to send to calling function
  var credentials = {
    url: S3_DOMAIN,
    key: filePath,
    AWSAccessKeyId: AWS_ACCESS_KEY_ID,
    acl: readType,
    policy: base64Policy,
    signature: signature,
    redirect: redirect,
    content_type: contentType,
    expiration: expiration
  };

  callback(null, credentials);
}

FRONTEND假设服务器的值在输入字段中,并且您通过表单提交提交图像(即POST,因为我无法让PUT工作):

function dataURItoBlob(dataURI, contentType) {
  var binary = atob(dataURI.split(',')[1]);
  var array = [];
  for(var i = 0; i < binary.length; i++) {
    array.push(binary.charCodeAt(i));
  }
  return new Blob([new Uint8Array(array)], {type: contentType});
}

function submitS3(callback) {
  var base64Data = $("#file").val();//your file to upload e.g. img.toDataURL("image/jpeg")
  var contentType = $("#contentType").val();
  var xmlhttp = new XMLHttpRequest();
  var blobData = dataURItoBlob(base64Data, contentType);

  var fd = new FormData();
  fd.append('key', $("#key").val());
  fd.append('acl', $("#acl").val());
  fd.append('Content-Type', contentType);
  fd.append('AWSAccessKeyId', $("#accessKeyId").val());
  fd.append('policy', $("#policy").val());
  fd.append('signature', $("#signature").val());
  fd.append("redirect", $("#redirect").val());
  fd.append("file", blobData);

  xmlhttp.onreadystatechange=function(){
    if (xmlhttp.readyState==4) {
      //do whatever you want on completion
      callback();
    }
  }
  var someBucket = "your_bucket_name"
  var S3_DOMAIN = "https://"+someBucket+".s3.amazonaws.com/";
  xmlhttp.open('POST', S3_DOMAIN, true);
  xmlhttp.send(fd);
}

注意:我每次提交时上传的图片超过1张,因此我添加了多个iframe(上面带有FRONTEND代码)来同时进行多图片上传.