如何通过 kubectl 在 google-cluoud-composer 和 Airflow 1.10 上访问 Airflow

Question

我正在使用GCP Composer来管理 GCP 上的 Apache。

对于新项目，我使用新版本的 Composer/Airflow（composer：1.6.1，Airflow：1.10）

要通过 shell 连接气流以检查损坏的 DAG，GCP 文档说明：

1. 打开 GCP shell

2. 连接到 GKE 集群

3. myuser @cloudshell:~ kubectl 获取 pod

4. myuser @cloudshell:~ kubectl exec -it airflow-worker- 1a2b3c - xyz12 -c airflow-worker -- /bin/bash

这在 Airflow 1.9 上工作正常，但在 Airflow 1.10 上kubectl get pods不显示工作 pod，而且我还没有找到有关如何在 AF 1.10 上通过 kubeclt 访问气流的文档

有人可以帮助我吗？

_myuser_@cloudshell:~ (_Myproject_)$ kubectl get pods 
NAME                                                        READY     STATUS      RESTARTS   AGE
airflow-monitoring-564c8c7dc5-hxb62                         1/1       Running     0          17h
airflow-redis-0                                             1/1       Running     0          17h
airflow-sqlproxy-594dbf87b7-nmtbh                           1/1       Running     0          17h
composer-agent-1a871e5e-fa97-4fa3-8843-d0b06718f7f6-rb4rj   0/1       Completed   0          12h
composer-agent-41ee85f8-90f2-45a1-a148-883421775651-jfjz6   0/1       Completed   0          12h
composer-agent-6984c0ef-4a0b-4150-bc6e-8a2996a5c38f-sd45f   0/1       Completed   0          17h
composer-agent-97de1623-5772-496d-a0c0-54adefc00c5a-rq9w6   0/1       Completed   0          12h
composer-agent-b3bc0dc6-6e95-4c9c-91d0-735755c2210f-tcxgg   0/1       Completed   0          73m
composer-fluentd-daemon-bkn28                               1/1       Running     133        17h
composer-fluentd-daemon-jvdlc                               1/1       Running     133        17h
composer-fluentd-daemon-lchq9                               1/1       Running     133        17h
composer-fluentd-daemon-wfzzr                               1/1       Running     133        17h

Answer 1

为了适应就地版本升级，Cloud Composer 现在在以运行版本命名的非默认命名空间中运行特定于版本的 Pod（工作线程和调度程序）。例如，它可能如下所示：

$ kubectl get ns | grep composer  # ns = namespaces
composer-1-6-1-airflow-1-10-1-04c210ec   Active   2d

然后，您可以专门在该命名空间中查找集群资源：

$ kubectl get pods --namespace=composer-1-6-1-airflow-1-10-1-04c210ec
NAME                                 READY   STATUS      RESTARTS   AGE
airflow-database-init-job-mhn87      0/1     Completed   0          2d
airflow-scheduler-748ff8dc88-g2k5q   2/2     Running     0          2d
airflow-worker-5767579d-csvcg        2/2     Running     0          2d
airflow-worker-5767579d-d8fgd        2/2     Running     0          2d
airflow-worker-5767579d-tlwpc        2/2     Running     0          2d

如果更方便的话，你还可以列出所有命名空间中的资源：

$ kubectl get pods --all-namespaces  # or -A

$ kubectl get pods -A | grep airflow
composer-1-6-1-airflow-1-10-1-04c210ec   airflow-database-init-job-mhn87                                  0/1     Completed   0          2d
composer-1-6-1-airflow-1-10-1-04c210ec   airflow-scheduler-748ff8dc88-g2k5q                               2/2     Running     0          2d
composer-1-6-1-airflow-1-10-1-04c210ec   airflow-worker-5767579d-csvcg                                    2/2     Running     0          2d
composer-1-6-1-airflow-1-10-1-04c210ec   airflow-worker-5767579d-d8fgd                                    2/2     Running     0          2d
composer-1-6-1-airflow-1-10-1-04c210ec   airflow-worker-5767579d-tlwpc                                    2/2     Running     0          2d
default                                  airflow-monitoring-f78cd5c75-2n727                               1/1     Running     0          2d
default                                  airflow-redis-0                                                  1/1     Running     0          2d
default                                  airflow-sqlproxy-57b7976d59-qnkm7                                1/1     Running     0          2d

有关如何通过 SSH 连接到工作节点或调度程序 Pod 之一的示例：

# Long form
$ kubectl exec \
   --stdin --tty \
   --namespace=composer-1-8-1-airflow-1-10-3-6e595c80 
   airflow-scheduler-58948b8956-4sxwt /bin/bash

# Short form
$ kubectl exec -it \
   -n composer-1-8-1-airflow-1-10-3-6e595c80
   airflow-scheduler-58948b8956-4sxwt bash