Terraform:如何使用命名空间在Google Cloud(GKE)上创建Kubernetes集群？

Question

我是在做一个可以做到以下事情的例子:

1. 通过Terraform在GKE上创建一个Kubernetes集群 google_container_cluster
2. ...并且继续在其中创建名称空间,我想通过 kubernetes_namespace

我不确定的是如何连接新创建的集群和命名空间定义.例如,在添加时google_container_node_pool,我可以做类似的事情,cluster = "${google_container_cluster.hosting.name}"但我没有看到类似的东西kubernetes_namespace.

Answer 1

从理论上讲,可以像在单个提供程序的上下文中引用资源或数据源一样,从K8S(或任何其他)提供程序中引用GCP提供程序中的资源.

provider "google" {
  region = "us-west1"
}

data "google_compute_zones" "available" {}

resource "google_container_cluster" "primary" {
  name = "the-only-marcellus-wallace"
  zone = "${data.google_compute_zones.available.names[0]}"
  initial_node_count = 3

  additional_zones = [
    "${data.google_compute_zones.available.names[1]}"
  ]

  master_auth {
    username = "mr.yoda"
    password = "adoy.rm"
  }

  node_config {
    oauth_scopes = [
      "https://www.googleapis.com/auth/compute",
      "https://www.googleapis.com/auth/devstorage.read_only",
      "https://www.googleapis.com/auth/logging.write",
      "https://www.googleapis.com/auth/monitoring"
    ]
  }
}

provider "kubernetes" {
  host = "https://${google_container_cluster.primary.endpoint}"
  username = "${google_container_cluster.primary.master_auth.0.username}"
  password = "${google_container_cluster.primary.master_auth.0.password}"
  client_certificate = "${base64decode(google_container_cluster.primary.master_auth.0.client_certificate)}"
  client_key = "${base64decode(google_container_cluster.primary.master_auth.0.client_key)}"
  cluster_ca_certificate = "${base64decode(google_container_cluster.primary.master_auth.0.cluster_ca_certificate)}"
}

resource "kubernetes_namespace" "n" {
  metadata {
    name = "blablah"
  }
}

但是在实践中,由于已知的核心错误打破了跨供应商的依赖性,它可能无法按预期工作,请参阅https://github.com/hashicorp/terraform/issues/12393和https://github.com/hashicorp/terraform/问题/ 4149分别.

替代解决方案是:

1. 使用2阶段应用并首先定位 GKE集群,然后是依赖于它的任何其他东西,即terraform apply -target=google_container_cluster.primary然后terraform apply
2. 从K8S配置中分离出GKE集群配置,为它们提供完全隔离的工作流,并通过远程状态连接它们.

/terraform-gke/main.tf

terraform {
  backend "gcs" {
    bucket  = "tf-state-prod"
    prefix  = "terraform/state"
  }
}

provider "google" {
  region = "us-west1"
}

data "google_compute_zones" "available" {}

resource "google_container_cluster" "primary" {
  name = "the-only-marcellus-wallace"
  zone = "${data.google_compute_zones.available.names[0]}"
  initial_node_count = 3

  additional_zones = [
    "${data.google_compute_zones.available.names[1]}"
  ]

  master_auth {
    username = "mr.yoda"
    password = "adoy.rm"
  }

  node_config {
    oauth_scopes = [
      "https://www.googleapis.com/auth/compute",
      "https://www.googleapis.com/auth/devstorage.read_only",
      "https://www.googleapis.com/auth/logging.write",
      "https://www.googleapis.com/auth/monitoring"
    ]
  }
}

output "gke_host" {
  value = "https://${google_container_cluster.primary.endpoint}"
}

output "gke_username" {
  value = "${google_container_cluster.primary.master_auth.0.username}"
}

output "gke_password" {
  value = "${google_container_cluster.primary.master_auth.0.password}"
}

output "gke_client_certificate" {
  value = "${base64decode(google_container_cluster.primary.master_auth.0.client_certificate)}"
}

output "gke_client_key" {
  value = "${base64decode(google_container_cluster.primary.master_auth.0.client_key)}"
}

output "gke_cluster_ca_certificate" {
  value = "${base64decode(google_container_cluster.primary.master_auth.0.cluster_ca_certificate)}"
}

这里我们通过outputs 公开所有必要的配置,并使用后端来存储状态,以及这些输出在远程位置,在这种情况下是GCS.这使我们能够在下面的配置中引用它.

/terraform-k8s/main.tf

data "terraform_remote_state" "foo" {
  backend = "gcs"
  config {
    bucket  = "tf-state-prod"
    prefix  = "terraform/state"
  }
}

provider "kubernetes" {
  host = "https://${data.terraform_remote_state.foo.gke_host}"
  username = "${data.terraform_remote_state.foo.gke_username}"
  password = "${data.terraform_remote_state.foo.gke_password}"
  client_certificate = "${base64decode(data.terraform_remote_state.foo.gke_client_certificate)}"
  client_key = "${base64decode(data.terraform_remote_state.foo.gke_client_key)}"
  cluster_ca_certificate = "${base64decode(data.terraform_remote_state.foo.gke_cluster_ca_certificate)}"
}

resource "kubernetes_namespace" "n" {
  metadata {
    name = "blablah"
  }
}

这里可能或可能不明显的是,在创建/更新任何K8S资源之前必须创建/更新集群(如果此类更新依赖于集群的更新).

采用第二种方法通常是可取的(即使当/如果错误不是因素并且交叉提供者参考有效),因为它减少了爆炸半径并且定义了更明确的责任.这种部署通常是(IMO)让1个人/团队负责管理集群,而另一个人/团队负责管理K8S资源.

虽然可能存在重叠 - 例如,运营商希望在新的GKE集群之上部署日志记录和监控基础架构,因此跨供应商依赖性旨在满足此类用例.出于这个原因,我建议订阅上面提到的GH问题.