如何使用 terraform 部署 EMR Terraform，这是一个简单的开箱即用的工作示例

Question

如何使用 terraform 部署 EMR Terraform，这是一个简单的开箱即用的工作示例

Rub*_*uck 3 amazon-emr apache-spark terraform

我正在使用 Terraform v0.14.5 并尝试官方Terraform 示例及其指定的版本控制：

terraform {
  required_providers {
    aws = {
      source = "hashicorp/aws"
      version = "3.25.0"
    }
  }
}

provider "aws" {
  region = var.region
}

Run Code Online (Sandbox Code Playgroud)

“解决”一些明显的错误之后

更改：从allow_all更改为allow_access（安全组的名称）

aws_security_group.allow_access.id

Run Code Online (Sandbox Code Playgroud)

更改：从allow_all更改为allow_access（安全组的名称）

使 cidr_blocks 成为一个列表

更新emr版本

aws_security_group.allow_access.id

cidr_blocks = [aws_vpc.main.cidr_block]

release_label = "emr-6.2.0"

Run Code Online (Sandbox Code Playgroud)

我成功启动并计划但未能申请

Error: Error waiting for EMR Cluster state to be "WAITING" or "RUNNING": TERMINATING: BOOTSTRAP_FAILURE: Master instance (i-07e34ac1b04ebde01) failed attempting to download bootstrap action 1 file from S3

Run Code Online (Sandbox Code Playgroud)

该错误似乎源于：

  bootstrap_action {
    path = "s3://elasticmapreduce/bootstrap-actions/run-if"
    name = "runif"
    args = ["instance.isMaster=true", "echo running on master node"]
  }

Run Code Online (Sandbox Code Playgroud)

所以我下载了文件

aws s3 cp s3://elasticmapreduce/bootstrap-actions/run-if .

Run Code Online (Sandbox Code Playgroud)

并在本地添加：


  bootstrap_action {
    path = "file://${path.module}/run-if"
//    path = "s3://elasticmapreduce/bootstrap-actions/run-if"

    name = "runif"
    args = ["instance.isMaster=true", "echo running on master node"]
  }

Run Code Online (Sandbox Code Playgroud)

这是整个代码：


terraform {
  required_providers {
    aws = {
      source = "hashicorp/aws"
      version = "3.25.0"
    }
  }
}

provider "aws" {
  region = var.region
}

resource "aws_emr_cluster" "cluster" {
  name          = "emr-test-arn"
  release_label = "emr-6.2.0"
  applications  = ["Spark", "Zeppelin"]

  ec2_attributes {
    subnet_id                         = aws_subnet.main.id
    emr_managed_master_security_group = aws_security_group.allow_access.id
    emr_managed_slave_security_group  = aws_security_group.allow_access.id
    instance_profile                  = aws_iam_instance_profile.emr_profile.arn
  }

  master_instance_group {
    instance_type = "m5.xlarge"
  }

  core_instance_group {
    instance_count = 1
    instance_type  = "m5.xlarge"
  }

  tags = {
    role     = "rolename"
    dns_zone = "env_zone"
    env      = "env"
    name     = "name-env"
  }

  bootstrap_action {
//    path = "s3://elasticmapreduce/bootstrap-actions/run-if"
    path = "file://${path.module}/run-if"
    name = "runif"
    args = ["instance.isMaster=true", "echo running on master node"]
  }

  configurations_json = <<EOF
  [
    {
      "Classification": "hadoop-env",
      "Configurations": [
        {
          "Classification": "export",
          "Properties": {
            "JAVA_HOME": "/usr/lib/jvm/java-1.8.0"
          }
        }
      ],
      "Properties": {}
    },
    {
      "Classification": "spark-env",
      "Configurations": [
        {
          "Classification": "export",
          "Properties": {
            "JAVA_HOME": "/usr/lib/jvm/java-1.8.0"
          }
        }
      ],
      "Properties": {}
    }
  ]
EOF

  service_role = aws_iam_role.iam_emr_service_role.arn
}

resource "aws_security_group" "allow_access" {
  name        = "allow_access"
  description = "Allow inbound traffic"
  vpc_id      = aws_vpc.main.id

  ingress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = [aws_vpc.main.cidr_block]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  depends_on = [aws_subnet.main]

  lifecycle {
    ignore_changes = [
      ingress,
      egress,
    ]
  }

  tags = {
    name = "emr_test"
  }
}

resource "aws_vpc" "main" {
  cidr_block           = "168.31.0.0/16"
  enable_dns_hostnames = true

  tags = {
    name = "emr_test"
  }
}

resource "aws_subnet" "main" {
  vpc_id     = aws_vpc.main.id
  cidr_block = "168.31.0.0/20"

  tags = {
    name = "emr_test"
  }
}

resource "aws_internet_gateway" "gw" {
  vpc_id = aws_vpc.main.id
}

resource "aws_route_table" "r" {
  vpc_id = aws_vpc.main.id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.gw.id
  }
}

resource "aws_main_route_table_association" "a" {
  vpc_id         = aws_vpc.main.id
  route_table_id = aws_route_table.r.id
}

###

# IAM Role setups

###

# IAM role for EMR Service
resource "aws_iam_role" "iam_emr_service_role" {
  name = "iam_emr_service_role"

  assume_role_policy = <<EOF
{
  "Version": "2008-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "elasticmapreduce.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF
}

resource "aws_iam_role_policy" "iam_emr_service_policy" {
  name = "iam_emr_service_policy"
  role = aws_iam_role.iam_emr_service_role.id

  policy = <<EOF
{
    "Version": "2012-10-17",
    "Statement": [{
        "Effect": "Allow",
        "Resource": "*",
        "Action": [
            "ec2:AuthorizeSecurityGroupEgress",
            "ec2:AuthorizeSecurityGroupIngress",
            "ec2:CancelSpotInstanceRequests",
            "ec2:CreateNetworkInterface",
            "ec2:CreateSecurityGroup",
            "ec2:CreateTags",
            "ec2:DeleteNetworkInterface",
            "ec2:DeleteSecurityGroup",
            "ec2:DeleteTags",
            "ec2:DescribeAvailabilityZones",
            "ec2:DescribeAccountAttributes",
            "ec2:DescribeDhcpOptions",
            "ec2:DescribeInstanceStatus",
            "ec2:DescribeInstances",
            "ec2:DescribeKeyPairs",
            "ec2:DescribeNetworkAcls",
            "ec2:DescribeNetworkInterfaces",
            "ec2:DescribePrefixLists",
            "ec2:DescribeRouteTables",
            "ec2:DescribeSecurityGroups",
            "ec2:DescribeSpotInstanceRequests",
            "ec2:DescribeSpotPriceHistory",
            "ec2:DescribeSubnets",
            "ec2:DescribeVpcAttribute",
            "ec2:DescribeVpcEndpoints",
            "ec2:DescribeVpcEndpointServices",
            "ec2:DescribeVpcs",
            "ec2:DetachNetworkInterface",
            "ec2:ModifyImageAttribute",
            "ec2:ModifyInstanceAttribute",
            "ec2:RequestSpotInstances",
            "ec2:RevokeSecurityGroupEgress",
            "ec2:RunInstances",
            "ec2:TerminateInstances",
            "ec2:DeleteVolume",
            "ec2:DescribeVolumeStatus",
            "ec2:DescribeVolumes",
            "ec2:DetachVolume",
            "iam:GetRole",
            "iam:GetRolePolicy",
            "iam:ListInstanceProfiles",
            "iam:ListRolePolicies",
            "iam:PassRole",
            "s3:CreateBucket",
            "s3:Get*",
            "s3:List*",
            "sdb:BatchPutAttributes",
            "sdb:Select",
            "sqs:CreateQueue",
            "sqs:Delete*",
            "sqs:GetQueue*",
            "sqs:PurgeQueue",
            "sqs:ReceiveMessage"
        ]
    }]
}
EOF
}

# IAM Role for EC2 Instance Profile
resource "aws_iam_role" "iam_emr_profile_role" {
  name = "iam_emr_profile_role"

  assume_role_policy = <<EOF
{
  "Version": "2008-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF
}

resource "aws_iam_instance_profile" "emr_profile" {
  name = "emr_profile"
  role = aws_iam_role.iam_emr_profile_role.name
}

resource "aws_iam_role_policy" "iam_emr_profile_policy" {
  name = "iam_emr_profile_policy"
  role = aws_iam_role.iam_emr_profile_role.id

  policy = <<EOF
{
    "Version": "2012-10-17",
    "Statement": [{
        "Effect": "Allow",
        "Resource": "*",
        "Action": [
            "cloudwatch:*",
            "dynamodb:*",
            "ec2:Describe*",
            "elasticmapreduce:Describe*",
            "elasticmapreduce:ListBootstrapActions",
            "elasticmapreduce:ListClusters",
            "elasticmapreduce:ListInstanceGroups",
            "elasticmapreduce:ListInstances",
            "elasticmapreduce:ListSteps",
            "kinesis:CreateStream",
            "kinesis:DeleteStream",
            "kinesis:DescribeStream",
            "kinesis:GetRecords",
            "kinesis:GetShardIterator",
            "kinesis:MergeShards",
            "kinesis:PutRecord",
            "kinesis:SplitShard",
            "rds:Describe*",
            "s3:*",
            "sdb:*",
            "sns:*",
            "sqs:*"
        ]
    }]
}
EOF
}

Run Code Online (Sandbox Code Playgroud)

谁能推荐一个可行的示例或帮助解决 VPC 错误吗？

Answer 1

Rub*_*uck 5

事实证明，我有一个新版本的 terraform CLI，即 14.5，它不适用于网络上的大多数示例。因此，对于非启动者（没有版本特殊知识的人......），我添加了如何启动和运行集群的特定“接收”。

我使用 https://github.com/cloudposse/terraform-aws-emr-cluster.git 因为它首先出现并持续维护。请记住，它使用了许多远程模块，尽管它们位于 Github 上，但它们具有版本依赖性，维护它们本身就是一个挑战。另请记住，就我而言，这只是一个“Hello World”。

安装 tfswitch。在苹果电脑上：

brew install tfswitch

tfswitch 0.13.5

Run Code Online (Sandbox Code Playgroud)

克隆 repo，将其放入 cd 并 init。我的工作使用了这个特定的提交：ed81e4259ae66178e6cbb7dcea75596f1701fe61，所以如果你需要检查它，这样你就可以有一个合理的起点。

git clone https://github.com/cloudposse/terraform-aws-emr-cluster.git
cd /terraform-aws-emr-cluster/examples/complete/
terraform init

Run Code Online (Sandbox Code Playgroud)

这将从 Github 下载源代码

通过复制和编辑文件来配置变量：

cp fixtures.us-east-2.tfvars terraform.tfvars

Run Code Online (Sandbox Code Playgroud)

创建一个 Secrets 目录并确保为其配置路径

mkdir <path of your choice>secrets

ssh_public_key_path = <path of your choice>secrets

Run Code Online (Sandbox Code Playgroud)

配置 EMR 集群：

terraform plan
terraform apply -auto-approve

Run Code Online (Sandbox Code Playgroud)

这应该会产生一个 EMR 集群。

聚苯乙烯

我想要的只是一个 POC 来测试架构解决方案。过去需要花 20 分钟才能完成的事情现在变得非常复杂，并且对于新手来说具有挑战性。基础设施即代码、声明性代码、简单性、干净的代码和简单理智的原则似乎因 DevOps Babylon Tower 和安全性的特性而受到损害。

归档时间：	4 年，8 月前
查看次数：	3279 次
最近记录：	4 年，8 月前