我们有32个V-CPU,RAM为28 GB,Local Executor
但是气流仍然在利用所有资源,这导致资源的过度利用,最终破坏了系统的执行。
以下是根据内存使用情况排序的ps -aux输出。
PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
1336 3.5 0.9 1600620 271644 ? Ss Feb18 23:41 /usr/bin/python /usr/local/bin/airflow webs
9434 32.3 0.9 1835796 267844 ? Sl 03:09 0:31 [ready] gunicorn: worker [airflow-webserver
10043 9.1 0.9 1835796 267844 ? Sl 03:05 0:33 [ready] gunicorn: worker [airflow-webserver
25397 17.4 0.9 1835796 267844 ? Sl 03:08 0:30 [ready] gunicorn: worker [airflow-webserver
30680 13.0 0.9 1835796 267844 ? Sl 03:06 0:36 [ready] gunicorn: worker …
Run Code Online (Sandbox Code Playgroud) 我正在尝试在Google Cloud Compute引擎实例上部署Airflow。
为了部署脚本,初始化操作中有一些特定的更改(用于初始化云VM的Shell脚本)。我想知道我是否可以使用Terraform处理此问题。
这是我的Terraform脚本。
provider "google" {
region = "${var.region}"
project = "${var.project_name}"
credentials = "${file("${var.credentials_file_path}")}"
zone = "${var.region_zone}"
}
resource "google_sql_database_instance" "master" {
name = "${var.db_instance}"
region = "${var.region}"
settings {
tier = "db-n1-standard-1"
}
}
resource "google_sql_user" "users" {
name = "${var.db_user}"
instance = "${google_sql_database_instance.master.name}"
host = "%"
password = "${var.db_password}"
depends_on = ["google_sql_database_instance.master"]
}
resource "google_sql_database" "airflow" {
name = "${var.db_name}"
instance = "${google_sql_database_instance.master.name}"
charset = "utf8"
collation = "utf8_general_ci"
depends_on = ["google_sql_database_instance.master"]
}
resource "google_compute_instance" …
Run Code Online (Sandbox Code Playgroud)