允许VPC中的Lambda访问同一VPC中的Elasticsearch域

Question

允许VPC中的Lambda访问同一VPC中的Elasticsearch域

non*_*com 6 amazon-ec2 amazon-web-services elasticsearch aws-cloudformation amazon-vpc

我正在学习如何使用 Amazon 服务，特别是目前我想使用 Cloud Formation 脚本创建一个简单的设置：一个带有用 JS 编写的单个 lambda 的 VPC，该 VPC 可以访问同一 VPC 中的 Elasticsearch 服务。

不知怎的，我无法让它工作。从 lambda 到 Elasticsearch 域的所有请求始终会超时。但是，从同一 VPC 中运行 Amazon Linux 2 的 EC2 实例中的相同 JS 代码或curl（即使没有任何额外授权，只需卷曲 ES 域终端节点）发出的相同请求可以正常工作，并且我可以与 Elasticsearch 正常通信来自该 EC2 实例（通过 SSH 连接到其中）。

同时，lambda 能够访问 VPC 中的 Aurora 集群，因此 lambda 无法访问 VPC 资源并不是一般问题。

请告诉我我在 Cloud Formation 的设置描述中做错了什么？以下是我的 Cloud Formation 模板的相关摘录以及能够从 EC2 实例访问 ES 服务的 JS 代码示例，但无法对 lambda 执行相同的操作：

AWSTemplateFormatVersion: 2010-09-09
Description: The AWS CloudFormation tutorial
Resources:

  SomeDeploymentBucket:
    Type: 'AWS::S3::Bucket'

  AppLogGroup:
    Type: 'AWS::Logs::LogGroup'
    Properties:
      LogGroupName: /aws/lambda/some-lambda

    # ========= The Lambda Execution Role =========

  IamRoleLambdaExecution:
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - lambda.amazonaws.com
            Action:
              - 'sts:AssumeRole'
      Policies:
        - PolicyName: !Join 
            - '-'
            - - dev
              - some-app
              - lambda
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - 's3:*'
                  - 'rds-db:connect'
                  - 'rds:*'
                  - 'es:*'
                Resource: '*'
      Path: /
      RoleName: !Join 
        - '-'
        - - some-app
          - dev
          - eu-west-1
          - lambdaRole
      ManagedPolicyArns:
        - !Join 
          - ''
          - - 'arn:'
            - !Ref 'AWS::Partition'
            - ':iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole'

    # ========= The Lambda =========

  AppLambdaFunction:
    Type: 'AWS::Lambda::Function'
    Properties:
      Code:
        S3Bucket: !Ref SomeDeploymentBucket
        S3Key: >-
          tutorial/some-app/dev/1545610972669-2018-12-24T00:22:52.669Z/some-app.zip
      FunctionName: some-lambda
      Handler: app.server
      MemorySize: 1024
      Role: !GetAtt 
        - IamRoleLambdaExecution
        - Arn
      Runtime: nodejs8.10
      Timeout: 6
      VpcConfig:
        SecurityGroupIds:
          - !Ref xxxVPCSecurityGroup
        SubnetIds:
          - !Ref xxxLambdaSubnet
    DependsOn:
      - AppLogGroup
      - IamRoleLambdaExecution

    # ========= VPC =========

  xxxVPC:
    Type: 'AWS::EC2::VPC'
    Properties:
      CidrBlock: 172.31.0.0/16
      InstanceTenancy: default
      EnableDnsSupport: 'true'
      EnableDnsHostnames: 'true'

  xxxVPCSecurityGroup:
    Type: 'AWS::EC2::SecurityGroup'
    Properties:
      GroupName: VPC SG
      GroupDescription: VPC Security Group
      VpcId: !Ref xxxVPC

  xxxLambdaSubnet:
    Type: 'AWS::EC2::Subnet'
    Properties:
      VpcId: !Ref xxxVPC
      CidrBlock: 172.31.32.0/20

    # ========= Elasticsearch =========

  xxxESSecurityGroup:
    Type: 'AWS::EC2::SecurityGroup'
    Properties:
      GroupName: ES SG
      GroupDescription: ES Security group
      VpcId: !Ref xxxVPC
      SecurityGroupIngress:
        - IpProtocol: -1
          FromPort: 0
          ToPort: 65535
          SourceSecurityGroupId: !Ref xxxVPCSecurityGroup

  xxxElasticSearch:
    Type: 'AWS::Elasticsearch::Domain'
    Properties:
      AccessPolicies:
        Version: 2012-10-17
        Statement:
          - Action:
              - 'es:*'
              - 'ec2:*'
              - 's3:*'
            Principal:
              AWS:
                - '*'
            Resource: '*'
            Effect: Allow
      DomainName: es-xxx-domain
      AdvancedOptions:
        rest.action.multi.allow_explicit_index: 'true'
      ElasticsearchVersion: 6.3
      ElasticsearchClusterConfig:
        InstanceCount: 2
        InstanceType: m3.medium.elasticsearch
        DedicatedMasterEnabled: 'false'
      VPCOptions:
        SecurityGroupIds:
          - !Ref xxxESSecurityGroup
        SubnetIds:
          - !Ref xxxLambdaSubnet

Run Code Online (Sandbox Code Playgroud)

JS代码（没有使用凭证签名的版本，但签名时也不起作用）：

var es = require('elasticsearch');
var client = new es.Client({
    host: 'vpc-es-domain-AMAZON.eu-west-1.es.amazonaws.com:80',
    log: 'trace'
});

client.ping({
    requestTimeout: 1000
}, function(error, res, status){
    if(error) {
        console.trace('es cluster error!');
        console.trace(error);
    } else {
        console.log('All is well');
        var response = {
            error: error,
            res: res,
            status: status
        }
        console.log(JSON.stringify(response));
    }
});

Run Code Online (Sandbox Code Playgroud)

在同一 VPC 中的 EC2 实例中执行此操作，可以毫无问题地获得来自 ES 域的响应：

curl vpc-es-domain-AMAZON.eu-west-1.es.amazonaws.com:80

Run Code Online (Sandbox Code Playgroud)

我真的很感激帮助，因为我已经被这个问题困扰了。

Answer 1

小智 3

注意到您的设置中有两个问题

您仅在堆栈中创建一个子网，并且仅将一个子网分配给 Lambda。您需要为 Lambda 分配多个子网
您需要修复 ES 的访问策略。我在控制台中进行测试时手动更新为“不需要使用 IAM 凭证签署请求”。

我更新了 Cloudformation 模板以创建基于 python 的 lambda 处理程序，以从同一 vpc 查询弹性搜索。它并不完整，但如果您确定上述问题，那么它应该可以工作。

AWSTemplateFormatVersion: 2010-09-09
Description: The AWS CloudFormation tutorial
Resources:

  SomeDeploymentBucket:
    Type: 'AWS::S3::Bucket'

  AppLogGroup:
    Type: 'AWS::Logs::LogGroup'
    Properties:
      LogGroupName: /aws/lambda/some-lambda

    # ========= The Lambda Execution Role =========

  IamRoleLambdaExecution:
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - lambda.amazonaws.com
            Action:
              - 'sts:AssumeRole'
      Policies:
        - PolicyName: !Join
            - '-'
            - - dev
              - some-app
              - lambda
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - 's3:*'
                  - 'rds-db:connect'
                  - 'rds:*'
                  - 'es:*'
                Resource: '*'


      Path: /
      RoleName: !Join
        - '-'
        - - some-app
          - dev
          - eu-west-1
          - lambdaRole
      ManagedPolicyArns:
        - !Join
          - ''
          - - 'arn:'
            - !Ref 'AWS::Partition'
            - ':iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole'

    # ========= The Lambda =========

  AppLambdaFunction:
    Type: AWS::Lambda::Function
    DependsOn:
      - AppLogGroup
      - IamRoleLambdaExecution
    Properties:
      FunctionName: some-lambda
      Handler: index.lambda_handler
      Runtime: python2.7
      Timeout: 60
      MemorySize: 1024
      Role: !GetAtt
        - IamRoleLambdaExecution
        - Arn
      VpcConfig:
        SecurityGroupIds:
          - !Ref xxxVPCSecurityGroup
        SubnetIds:
          - !Ref xxxLambdaSubnet1
          - !Ref xxxLambdaSubnet2

      Code:
        ZipFile: !Sub |
          from __future__ import print_function
          import boto3
          iam = boto3.client('iam')

          def lambda_handler(event, context):
            print('called lambda_handler')

    # ========= VPC =========

  xxxVPC:
    Type: 'AWS::EC2::VPC'
    Properties:
      CidrBlock: 172.31.0.0/16
      InstanceTenancy: default
      EnableDnsSupport: 'true'
      EnableDnsHostnames: 'true'

  xxxVPCSecurityGroup:
    Type: 'AWS::EC2::SecurityGroup'
    Properties:
      GroupName: VPC SG
      GroupDescription: VPC Security Group
      VpcId: !Ref xxxVPC

  xxxLambdaSubnet1:
    Type: 'AWS::EC2::Subnet'
    Properties:
      VpcId: !Ref xxxVPC
      CidrBlock: 172.31.32.0/20
  xxxLambdaSubnet2:
    Type: 'AWS::EC2::Subnet'
    Properties:
      VpcId: !Ref xxxVPC
      CidrBlock: 172.31.16.0/20


    # ========= Elasticsearch =========

  xxxESSecurityGroup:
    Type: 'AWS::EC2::SecurityGroup'
    Properties:
      GroupName: ES SG
      GroupDescription: ES Security group
      VpcId: !Ref xxxVPC
      SecurityGroupIngress:
        - IpProtocol: -1
          FromPort: 0
          ToPort: 65535
          SourceSecurityGroupId: !Ref xxxVPCSecurityGroup

  xxxElasticSearch:
    Type: 'AWS::Elasticsearch::Domain'
    Properties:
      AccessPolicies:
        Version: 2012-10-17
        Statement:
          - Action:
              - 'es:*'
              - 'ec2:*'
              - 's3:*'
            Principal:
              AWS:
                - '*'
            Resource: '*'
            Effect: Allow
      DomainName: es-xxx-domain
      EBSOptions:
        EBSEnabled: true
        Iops: 0
        VolumeSize: 20
        VolumeType: "gp2"
      AdvancedOptions:
        rest.action.multi.allow_explicit_index: 'true'

      ElasticsearchClusterConfig:
        InstanceCount: 1
        InstanceType: m4.large.elasticsearch
        DedicatedMasterEnabled: 'false'
      VPCOptions:
        SecurityGroupIds:
          - !Ref xxxESSecurityGroup
        SubnetIds:
          - !Ref xxxLambdaSubnet1

Run Code Online (Sandbox Code Playgroud)

更新了 Lambda 函数处理程序代码

  import urllib2

def lambda_handler(event, context):
    print('called lambda_handler')
    data = ''
    url = 'https://vpc-es-xxx-domain-fixthis.es.amazonaws.com'
    req = urllib2.Request(url, data, {'Content-Type': 'application/json'})
    f = urllib2.urlopen(req)
    for x in f:
        print(x)
    f.close()

Run Code Online (Sandbox Code Playgroud)

归档时间：	6 年，10 月前
查看次数：	7733 次
最近记录：	6 年，3 月前