从Amazon S3存储桶下载文件的脚本

Question

尝试编写脚本以从Amazon S3存储桶下载文件.

在cURL网站上遇到示例问题.下面的脚本产生:

我们计算的请求签名与您提供的签名不匹配.检查您的密钥和签名方法.

感谢任何帮助.

#!/bin/sh 
file="filename.php"
bucket="my-bucket"
resource="/${bucket}/${file}"
contentType="text/html"
dateValue="`date +'%a, %d %b %Y %H:%M:%S %z'`"
stringToSign="GET\n\n${contentType}\n${dateValue}\n${resource}"
s3Key='ABCABCABCABCABCABCAB'
s3Secret='xyzxyzyxyzxyzxyzxyzxyzxyzxyzxyzxyzxyzxyzx'
signature=`/bin/echo -en "$stringToSign" | openssl sha1 -hmac ${s3Secret} -      binary | base64`

curl -v -H "Host:lssngen-updates-east.s3.amazonaws.com" \
        -H "Date:${dateValue}" \
        -H "Content-Type:${contentType}" \
        -H "Authorization: AWS ${s3Key}:${signature}" \
        https://${bucket}.s3.amazonaws.com/${file}

Answer 1

避免自己签署请求,很多可能出错或难以做到.例如,您应该检查日期是否设置为GMT或使用x-amz-date标头.

另一种方法是使用AWS命令行界面,因此使用 $ aws s3 cp或$ aws s3 sync.

Answer 2

我写这个bash脚本从s3下载文件(我下载压缩文件,你可以改变contentType来下载其他类型的文件)

#!/bin/sh 
outputFile="Your_PATH"
amzFile="AMAZON_FILE_PATH"
bucket="YOUR_BUCKET"
resource="/${bucket}/${amzFile}"
contentType="application/x-compressed-tar"
dateValue=`date -R`
stringToSign="GET\n\n${contentType}\n${dateValue}\n${resource}"
s3Key="YOUR_S3_KEY"
s3Secret="YOUR_S3SECRET"
signature=`echo -en ${stringToSign} | openssl sha1 -hmac ${s3Secret} -binary | base64`

curl  -H "Host: ${bucket}.s3.amazonaws.com" \
     -H "Date: ${dateValue}" \
     -H "Content-Type: ${contentType}" \
     -H "Authorization: AWS ${s3Key}:${signature}" \
     https://${bucket}.s3.amazonaws.com/${amzFile} -o $outputFile

Answer 3

截至 2019 年 8 月，我发现这很有效。已添加区域，并且 URL 格式已更改。

#!/bin/sh 
outputFile="/PATH/TO/LOCALLY/SAVED/FILE"
amzFile="BUCKETPATH/TO/FILE"
region="YOUR-REGION"
bucket="SOME-BUCKET"
resource="/${bucket}/${amzFile}"
contentType="binary/octet-stream"
dateValue=`TZ=GMT date -R`
# You can leave our "TZ=GMT" if your system is already GMT (but don't have to)
stringToSign="GET\n\n${contentType}\n${dateValue}\n${resource}"
s3Key="ACCESS_KEY_ID"
s3Secret="SECRET_ACCESS_KEY"
signature=`echo -en ${stringToSign} | openssl sha1 -hmac ${s3Secret} -binary | base64`
curl -H "Host: s3-${region}.amazonaws.com" \
     -H "Date: ${dateValue}" \
     -H "Content-Type: ${contentType}" \
     -H "Authorization: AWS ${s3Key}:${signature}" \
     https://s3-${region}.amazonaws.com/${bucket}/${amzFile} -o $outputFile

Answer 4

一个真正有效的解决方案（将于 2021 年 12 月开始工作）是使用此脚本。只需在使用前导出密钥（或将其值复制到 .sh 文件）。

export AWS_ACCESS_KEY_ID=AKxxx
export AWS_SECRET_ACCESS_KEY=zzzz

下载只需运行

./s3download.sh get s3://mybucket/myfile.txt myfile.txt

您只需传递get、s3 存储桶以及文件名和输出文件。

需要脚本

创建一个s3download.sh文件 和chmod +x s3download.sh，以在上面的命令中使用。

#!/bin/bash
set -eu
s3simple() {
  local command="$1"
  local url="$2"
  local file="${3:--}"

  # todo: nice error message if unsupported command?

  if [ "${url:0:5}" != "s3://" ]; then
    echo "Need an s3 url"
    return 1
  fi
  local path="${url:4}"

  if [ -z "${AWS_ACCESS_KEY_ID-}"  ]; then
    echo "Need AWS_ACCESS_KEY_ID to be set"
    return 1
  fi

  if [ -z "${AWS_SECRET_ACCESS_KEY-}" ]; then
    echo "Need AWS_SECRET_ACCESS_KEY to be set"
    return 1
  fi

  local method md5 args
  case "$command" in
  get)
    method="GET"
    md5=""
    args="-o $file"
    ;;
  put)
    method="PUT"
    if [ ! -f "$file" ]; then
      echo "file not found"
      exit 1
    fi
    md5="$(openssl md5 -binary $file | openssl base64)"
    args="-T $file -H Content-MD5:$md5"
    ;;
  *)
    echo "Unsupported command"
    return 1
  esac

  local date="$(date -u '+%a, %e %b %Y %H:%M:%S +0000')"
  local string_to_sign
  printf -v string_to_sign "%s\n%s\n\n%s\n%s" "$method" "$md5" "$date" "$path"
  local signature=$(echo -n "$string_to_sign" | openssl sha1 -binary -hmac "${AWS_SECRET_ACCESS_KEY}" | openssl base64)
  local authorization="AWS ${AWS_ACCESS_KEY_ID}:${signature}"

  curl $args -s -f -H Date:"${date}" -H Authorization:"${authorization}" https://s3.amazonaws.com"${path}"
}

s3simple "$@"

您可以在此处找到有关 s3simple 脚本的更多信息。