AWS CDK - 如何向现有 S3 存储桶添加事件通知

Question

我正在尝试修改此AWS 提供的 CDK 示例以改为使用现有存储桶。其他文档表明支持导入现有资源。到目前为止，我无法使用 CDK 向现有存储桶添加事件通知。

这是我修改后的示例版本：

class S3TriggerStack(core.Stack):

    def __init__(self, scope: core.Construct, id: str, **kwargs) -> None:
        super().__init__(scope, id, **kwargs)

        # create lambda function
        function = _lambda.Function(self, "lambda_function",
                                    runtime=_lambda.Runtime.PYTHON_3_7,
                                    handler="lambda-handler.main",
                                    code=_lambda.Code.asset("./lambda"))

        # **MODIFIED TO GET EXISTING BUCKET**
        #s3 = _s3.Bucket(self, "s3bucket")
        s3 = _s3.Bucket.from_bucket_arn(self, 's3_bucket',
            bucket_arn='arn:<my_region>:::<my_bucket>')

        # create s3 notification for lambda function
        notification = aws_s3_notifications.LambdaDestination(function)

        # assign notification for the s3 event type (ex: OBJECT_CREATED)
        s3.add_event_notification(_s3.EventType.OBJECT_CREATED, notification)

这会在尝试时导致以下错误add_event_notification：

AttributeError: '_IBucketProxy' object has no attribute 'add_event_notification'

该from_bucket_arn函数返回一个IBucket，该add_event_notification函数是Bucket该类的一个方法，但我似乎找不到任何其他方法来做到这一点。可能不支持。任何帮助，将不胜感激。

Answer 1

自 2021 年 6 月以来，有更好的方法来解决这个问题。由于大约。CDK 版本 1.110.0 可以将 S3 通知与 Typescript 代码一起使用：

例子：

const s3Bucket = s3.Bucket.fromBucketName(this, 'bucketId', 'bucketName');
s3Bucket.addEventNotification(s3.EventType.OBJECT_CREATED, new s3n.LambdaDestination(lambdaFunction), {
    prefix: 'example/file.txt'
});

CDK 文档： https://docs.aws.amazon.com/cdk/api/latest/docs/aws-s3-notifications-readme.html

拉取请求： https ://github.com/aws/aws-cdk/pull/15158

Answer 2

我设法使用自定义资源使其工作。它是 TypeScript，但它应该很容易转换为 Python：

const uploadBucket = s3.Bucket.fromBucketName(this, 'BucketByName', 'existing-bucket');

const fn = new lambda.Function(this, 'MyFunction', {
    runtime: lambda.Runtime.NODEJS_10_X,
    handler: 'index.handler',
    code: lambda.Code.fromAsset(path.join(__dirname, 'lambda-handler'))
});

const rsrc = new AwsCustomResource(this, 'S3NotificationResource', {
    onCreate: {
        service: 'S3',
        action: 'putBucketNotificationConfiguration',
        parameters: {
            // This bucket must be in the same region you are deploying to
            Bucket: uploadBucket.bucketName,
            NotificationConfiguration: {
                LambdaFunctionConfigurations: [
                    {
                        Events: ['s3:ObjectCreated:*'],
                        LambdaFunctionArn: fn.functionArn,
                        Filter: {
                            Key: {
                                FilterRules: [{ Name: 'suffix', Value: 'csv' }]
                            }
                        }
                    }
                ]
            }
        },
        // Always update physical ID so function gets executed
        physicalResourceId: 'S3NotifCustomResource' + Date.now().toString()
    }
});

fn.addPermission('AllowS3Invocation', {
    action: 'lambda:InvokeFunction',
    principal: new iam.ServicePrincipal('s3.amazonaws.com'),
    sourceArn: uploadBucket.bucketArn
});

rsrc.node.addDependency(fn.permissionsNode.findChild('AllowS3Invocation'));

这基本上是本示例中布局的 CloudFormation 模板的 CDK 版本。有关可能的参数，请参阅AWS 开发工具包上的文档NotificationConfiguration。

Answer 3

抱歉，由于声誉较低，我无法对上述出色的James Irwin 回答发表评论，但我将其制作成Construct.

关于“拒绝访问”的评论也花了我一些时间来弄清楚，但关键是函数是S3:putBucketNotificationConfiguration，但允许的 IAM 策略操作是S3:PutBucketNotification。

这是[构造代码]:( https://gist.github.com/archisgore/0f098ae1d7d19fddc13d2f5a68f606ab )

import * as cr from '@aws-cdk/custom-resources';
import * as logs from '@aws-cdk/aws-logs';
import * as s3 from '@aws-cdk/aws-s3';
import * as sqs from '@aws-cdk/aws-sqs';
import * as iam from '@aws-cdk/aws-iam';
import {Construct} from '@aws-cdk/core';

// You can drop this construct anywhere, and in your stack, invoke it like this:
// const s3ToSQSNotification = new S3NotificationToSQSCustomResource(this, 's3ToSQSNotification', existingBucket, queue);

export class S3NotificationToSQSCustomResource extends Construct {

    constructor(scope: Construct, id: string, bucket: s3.IBucket, queue: sqs.Queue) {
        super(scope, id);

        // https://stackoverflow.com/questions/58087772/aws-cdk-how-to-add-an-event-notification-to-an-existing-s3-bucket
        const notificationResource = new cr.AwsCustomResource(scope, id+"CustomResource", {
            onCreate: {
                service: 'S3',
                action: 'putBucketNotificationConfiguration',
                parameters: {
                    // This bucket must be in the same region you are deploying to
                    Bucket: bucket.bucketName,
                    NotificationConfiguration: {
                        QueueConfigurations: [
                            {
                                Events: ['s3:ObjectCreated:*'],
                                QueueArn: queue.queueArn,
                            }
                        ]
                    }
                },
                physicalResourceId: <cr.PhysicalResourceId>(id + Date.now().toString()),
            },
            onDelete: {
                service: 'S3',
                action: 'putBucketNotificationConfiguration',
                parameters: {
                    // This bucket must be in the same region you are deploying to
                    Bucket: bucket.bucketName,
                    // deleting a notification configuration involves setting it to empty.
                    NotificationConfiguration: {
                    }
                },
                physicalResourceId: <cr.PhysicalResourceId>(id + Date.now().toString()),
            },
            policy: cr.AwsCustomResourcePolicy.fromStatements([new iam.PolicyStatement({
                // The actual function is PutBucketNotificationConfiguration.
                // The "Action" for IAM policies is PutBucketNotification.
                // https://docs.aws.amazon.com/AmazonS3/latest/dev/list_amazons3.html#amazons3-actions-as-permissions
                actions: ["S3:PutBucketNotification"],
                 // allow this custom resource to modify this bucket
                resources: [bucket.bucketArn],
            })]),
            logRetention: logs.RetentionDays.ONE_DAY,
        });

        // allow S3 to send notifications to our queue
        // https://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html#grant-destinations-permissions-to-s3
        queue.addToResourcePolicy(new iam.PolicyStatement({
            principals: [new iam.ServicePrincipal("s3.amazonaws.com")],
            actions: ["SQS:SendMessage"],
            resources: [queue.queueArn],
            conditions: {
                ArnEquals: {"aws:SourceArn": bucket.bucketArn}
            }
        }));

        // don't create the notification custom-resource until after both the bucket and queue
        // are fully created and policies applied.
        notificationResource.node.addDependency(bucket);
        notificationResource.node.addDependency(queue);
    }
}

Answer 4

更新：原始答案的源代码将覆盖存储桶的现有通知列表，这将导致无法添加新的 lambda 触发器。这是使用事件源处理上述问题的解决方案。

import aws_cdk {
    aws_s3 as s3,
    aws_cdk.aws_lambda as lambda_
    aws_lambda_event_sources as event_src
}
import path as path

class S3LambdaTrigger(core.Stack):
    
    def __init__(self, scope: core.Construct, id: str):
        
        super().__init__(scope, id)
        
        bucket = s3.Bucket(
            self, "S3Bucket",
            block_public_access=s3.BlockPublicAccess.BLOCK_ALL,
            bucket_name='BucketName',
            encryption=s3.BucketEncryption.S3_MANAGED,
            versioned=True
        )

        fn = lambda_.Function(
            self, "LambdaFunction",
            runtime=lambda_.Runtime.NODEJS_10_X,
            handler="index.handler",
            code=lambda_.Code.from_asset(path.join(__dirname, "lambda-handler"))
        )

        fn.add_permission(
            's3-service-principal', 
            principal=aws_iam.ServicePrincipal('s3.amazonaws.com')
        )

        fn.add_event_source(
            event_src.S3EventSource(
                bucket, 
                events=[s3.EventType.OBJECT_CREATED, s3.EventType.OBJECT_REMOVED],
                filters=[s3.NotificationKeyFilter(prefix="subdir/", suffix=".txt")]
            )
        )

原文： 我在 TypeScript 中采用了ubi的解决方案并成功将其翻译成 Python。他的解决方案对我有用。

#!/usr/bin/env python

from typing import List

from aws_cdk import (
    core,
    custom_resources as cr,
    aws_lambda as lambda_,
    aws_s3 as s3,
    aws_iam as iam,
)


class S3NotificationLambdaProps:
    def __init__(self, bucket: s3.Bucket, function: lambda_.Function, events: List[str], prefix: str):
        self.bucket = bucket
        self.function = function
        self.events = events
        self.prefix = prefix


class S3NotificationLambda(core.Construct):

    def __init__(self, scope: core.Construct, id: str, props: S3NotificationLambdaProps):
        super().__init__(scope, id)

        self.notificationResource = cr.AwsCustomResource(
            self, f'CustomResource{id}',
            on_create=cr.AwsSdkCall(
                service="S3",
                action="S3:putBucketNotificationConfiguration",
                # Always update physical ID so function gets executed
                physical_resource_id=cr.PhysicalResourceId.of(f'S3NotifCustomResource{id}'),
                parameters={
                    "Bucket": props.bucket.bucket_name,
                    "NotificationConfiguration": {
                        "LambdaFunctionConfigurations": [{
                            "Events": props.events,
                            "LambdaFunctionArn": props.function.function_arn,
                            "Filter": {
                                "Key": {"FilterRules": [{"Name": "prefix", "Value": props.prefix}]}
                            }}
                        ]
                    }
                }
            ),
            on_delete=cr.AwsSdkCall(
                service="S3",
                action="S3:putBucketNotificationConfiguration",
                # Always update physical ID so function gets executed
                physical_resource_id=cr.PhysicalResourceId.of(f'S3NotifCustomResource{id}'),
                parameters={
                    "Bucket": props.bucket.bucket_name,
                    "NotificationConfiguration": {},
                }
            ),
            policy=cr.AwsCustomResourcePolicy.from_statements(
                statements=[
                    iam.PolicyStatement(
                        actions=["S3:PutBucketNotification", "S3:GetBucketNotification"],
                        resources=[props.bucket.bucket_arn]
                    ),
                ]
            )
        )

        props.function.add_permission(
            "AllowS3Invocation",
            action="lambda:InvokeFunction",
            principal=iam.ServicePrincipal("s3.amazonaws.com"),
            source_arn=props.bucket.bucket_arn,
        )

        # don't create the notification custom-resource until after both the bucket and lambda
        # are fully created and policies applied.
        self.notificationResource.node.add_dependency(props.bucket)
        self.notificationResource.node.add_dependency(props.function)

# Usage:

s3NotificationLambdaProps = S3NotificationLambdaProps(
    bucket=bucket_,
    function=lambda_fn_,
    events=['s3:ObjectCreated:*'],
    prefix='foo/'
)

s3NotificationLambda = S3NotificationLambda(
    self, "S3NotifLambda",
    self.s3NotificationLambdaProps
)