如何创建签名的S3 URL

Question

我想fread从data.table包中读取S3中的csv文件,如下所示:

 ulr_with_signature <- signURL(url, access_key, secret_key)
 DT <- fread(ulr_with_signature)

是否有一个包或一段代码允许我使用访问/密钥对构建URL.

我不想awscli用于阅读数据.

Answer 1

您可以使用AWS S3包:

执行阅读:

# These variables should be set in your environment, but you could set them in R:
Sys.setenv("AWS_ACCESS_KEY_ID" = "mykey",
       "AWS_SECRET_ACCESS_KEY" = "mysecretkey",
       "AWS_DEFAULT_REGION" = "us-east-1")

library("aws.s3")

如果您有obj要保存到AWS 的R对象,稍后会阅读:

s3save(obj, bucket = "my_bucket", object = "object")
# and then later
obj <- s3load("object", bucket = "my_bucket")

显然,将桶名称和文件名(AWS桶中的对象名称)替换为实际值.该包还具有相应的s3save功能.您还可以保存和RDS格式的负载s3saveRDS和s3readRDS.

如果你需要阅读一个文本文件,它会有点复杂,因为库的函数'get_object'返回一个原始向量,我们必须自己解析它:

raw_data <- get_object('data.csv', 'my_bucket')

# this method to parse the data is copied from the httr library
# substitute encoding from as needed
data <- iconv(readBin(raw_data, character()), from="UTF-8", to="UTF-8")

# now the data can be read by any R function, eg.
read.csv(data)
fread(data)

# All this can be done without temporary objects:
fread(iconv(
  readBin(get_object('data.csv', 'my_bucket'), character()),
  from="UTF-8", to="UTF-8"))

据我所知,您无法获得"签名网址"的概念.需要注意的是,您是否应该尝试开发这样的解决方案:考虑将秘密访问密钥存储在源代码中的安全隐患非常重要.

关于'signed url'的另一个问题是该对象将存储在内存中.如果保存工作空间,它将存储在磁盘上.这样的解决方案必须仔细审查安全性.