如何使用 AWS S3 put 对象解决减速错误

Question

我正在尝试访问一个文件并使用 boto 在 S3 中更新它，但即使按照下面的代码在请求之间暂停后，仍然会继续出现减速错误。我该如何解决这个问题？

body = b'Here we have some more data'
s3.put_object(Body=body,Bucket=bucket, Key=key)
time.sleep(10)
response = s3.get_object(Bucket=bucket, Key=key)
time.sleep(10)
print(response["Body"].read().decode('utf-8'))
currFile = response["Body"].read().decode('utf-8')
newFile = currFile + "\n" + "New Stuff!!!"
newFileB = newFile.encode('utf-8')
time.sleep(60)
s3.put_object(Body=newFileB,Bucket=bucket, Key=key)
time.sleep(10)
response = s3.get_object(Bucket=bucket, Key=key)
print(response["Body"].read().decode('utf-8'))

这是错误：

Details
The area below shows the result returned by your function execution.
{
"errorMessage": "An error occurred (SlowDown) when calling the PutObject operation (reached max retries: 4): Please reduce your request rate.",
"errorType": "ClientError",
"stackTrace": [
[
"/var/task/lambda_function.py",
43,
"lambda_handler",
"raise e"
],
[
"/var/task/lambda_function.py",
20,
"lambda_handler",
"s3.put_object(Body=body,Bucket=bucket, Key=key)"
],
[
"/var/runtime/botocore/client.py",
314,
"_api_call",
"return self._make_api_call(operation_name, kwargs)"
],
[
"/var/runtime/botocore/client.py",
612,
"_make_api_call",
"raise error_class(parsed_response, operation_name)"
]
]
}

Answer 1

我遇到了这个确切的问题，我不确定为什么会发生这种情况，但这是生产代码必须始终处理的事情。解决方案是继续尝试，并且在相当长的一段时间内不要放弃。发生故障时，循环会以 1 秒的延迟开始，然后在每个循环中将延迟增加一秒 (delay_incr)，最后每个循环的 max_delay 超时 30 秒，这实际上是最大总计 7.5 分钟，当最后放弃了。当然，你可以调整时间。到目前为止，这对我来说是成功的。

即使对于 NAS 文件服务器，我有时也必须等待一段时间才能读取文件，我也必须做类似的事情。

def put_s3_core(bucket: str, key: str, strobj: Any, content_type: str=''):  
    """ write strobj to s3 bucket, key
        content_type can be:
            binary/octet-stream (default)
            text/plain
            text/html
            text/csv
            image/png
            image/tiff
            application/pdf
            application/zip
            
    """
    if not strobj: 
        return
    
    delay = 0.5     # initial delay
    delay_incr = .5 # additional delay in each loop
    max_delay = 5   # max delay of one loop. Total delay is (max_delay**2)/2
    error_data = None
    
    while delay < max_delay:   
        try:
            s3 = boto3.resource('s3')
            request_obj = s3.Object(bucket, key)
            if content_type:
                request_obj.put(Body=strobj, ContentType=content_type)        
            else:
                request_obj.put(Body=strobj)
            break               
                
        except ClientError as err:
            code = err.response.get('Error',{}).get('Code', '')
            if code in ['NoSuchBucket']:
                print(f"Error: {code}. Check s3path: s3://{bucket}/{key}")
                raise err
            time.sleep(delay)
            delay += delay_incr
            error_data = err
    else:
        print(f"{logs.prog_loc()} Timeout encountered trying to write to s3://{bucket}/{key} strobj of len {len(strobj):,} bytes Error {error_data}")
        logs.error_beep()
        import pdb; pdb.set_trace() #perm
        
        raise TimeoutError

    ```