使用python删除S3中对象的所有版本？

Question

我有一个版本化的存储桶,并希望从存储桶中删除该对象(及其所有版本).但是,当我尝试从控制台删除对象时,S3只是添加删除标记但不执行硬删除.

是否可以使用特定键删除对象的所有版本(硬删除)？

s3resource = boto3.resource('s3')
bucket = s3resource.Bucket('my_bucket')
obj = bucket.Object('my_object_key')

# I would like to delete all versions for the object like so:
obj.delete_all_versions()

# or delete all versions for all objects like so:
bucket.objects.delete_all_versions()

Answer 1

我在使用其他解决方案来解决这个问题时遇到了麻烦，所以这是我的。

import boto3
bucket = "bucket name goes here"
filename = "filename goes here"

client = boto3.client('s3')
paginator = client.get_paginator('list_object_versions')
response_iterator = paginator.paginate(Bucket=bucket)
for response in response_iterator:
    versions = response.get('Versions', [])
    versions.extend(response.get('DeleteMarkers', []))
    for version_id in [x['VersionId'] for x in versions
                       if x['Key'] == filename and x['VersionId'] != 'null']:
        print('Deleting {} version {}'.format(filename, version_id))
        client.delete_object(Bucket=bucket, Key=filename, VersionId=version_id)

此代码处理以下情况

• 对象版本控制实际上并未打开
• 有DeleteMarker s
• 没有删除标记
• 给定文件的版本比单个 API 响应中的多

Mahesh Mogal 的回答不会删除DeleteMarkers。如果对象缺少.Mangohero1 的回答失败DeleteMarker。Hari 的回答重复了 10 次（以解决缺少分页逻辑的问题）。

Answer 2

该文档在这里很有帮助：

1. 在 S3 存储桶中启用版本控制时，简单的删除对象请求无法从该存储桶中永久删除对象。相反，Amazon S3 会插入删除标记（这实际上是具有自己的版本 ID 的对象的新版本）。
2. 当您尝试 GET 当前版本为删除标记的对象时，S3 的行为就像该对象已被删除（即使尚未删除）并返回 404 错误。
3. 要从版本化存储桶中永久删除对象，请对对象的每个版本（包括删除标记）使用带有相关版本 ID 的 DeleteObject。

Answer 3

其他答案分别删除对象。使用delete_objects boto3 调用和批处理您的删除更有效。请参阅下面的代码以获取一个函数，该函数以 1000 个为一组收集所有对象并删除：

bucket = 'bucket-name'
s3_client = boto3.client('s3')
object_response_paginator = s3_client.get_paginator('list_object_versions')

delete_marker_list = []
version_list = []

for object_response_itr in object_response_paginator.paginate(Bucket=bucket):
    if 'DeleteMarkers' in object_response_itr:
        for delete_marker in object_response_itr['DeleteMarkers']:
            delete_marker_list.append({'Key': delete_marker['Key'], 'VersionId': delete_marker['VersionId']})

    if 'Versions' in object_response_itr:
        for version in object_response_itr['Versions']:
            version_list.append({'Key': version['Key'], 'VersionId': version['VersionId']})

for i in range(0, len(delete_marker_list), 1000):
    response = s3_client.delete_objects(
        Bucket=bucket,
        Delete={
            'Objects': delete_marker_list[i:i+1000],
            'Quiet': True
        }
    )
    print(response)

for i in range(0, len(version_list), 1000):
    response = s3_client.delete_objects(
        Bucket=bucket,
        Delete={
            'Objects': version_list[i:i+1000],
            'Quiet': True
        }
    )
    print(response)

Answer 4

您可以使用 object_versions。

def delete_all_versions(bucket_name: str, prefix: str):
    s3 = boto3.resource('s3')
    bucket = s3.Bucket(bucket_name)
    if prefix is None:
        bucket.object_versions.delete()
    else:
        bucket.object_versions.filter(Prefix=prefix).delete()

delete_all_versions("my_bucket", None) # empties the entire bucket
delete_all_versions("my_bucket", "my_prefix/") # deletes all objects matching the prefix (can be only one if only one matches)

Answer 5

要删除某个前缀下的一个或多个对象的所有版本：

将对象键/folder/filename或前缀/folder/subfolder/传递给Prefix

import boto3

s3 = boto3.resource('s3')
bucket = s3.Bucket("my-bucket-name")
bucket.object_versions.filter(Prefix="folder/subfolder/").delete()

Answer 6

作为@jarmod答案的补充，这是我开发的一种解决方法来“硬删除”对象（包括删除标记的对象）；

def get_all_versions(bucket, filename):
    s3 = boto3.client('s3')
    keys = ["Versions", "DeleteMarkers"]
    results = []
    for k in keys:
        response = s3.list_object_versions(Bucket=bucket)[k]
        to_delete = [r["VersionId"] for r in response if r["Key"] == filename]
    results.extend(to_delete)
    return results

bucket = "YOUR BUCKET NAME"
file = "YOUR FILE"

for version in get_all_versions(bucket, file):
    s3.delete_object(Bucket=bucket, Key=file, VersionId=version)