通过 Airflow 创建的 Kubernetes pod 保持运行状态

Question

我已经在 Kubernetes 集群中设置了 Airflow。为了运行任务，我使用KubernetesPodOperator.

当我运行一个任务并查看 时kubectl get pods，我看到一个 pod 被正确创建并且它也完成了。然而，当我查看 Airflow 时，我看到状态没有更新，它说它仍然处于运行状态。

[2019-01-27 12:43:56,580] {models.py:1595} INFO - Executing <Task(KubernetesPodOperator): xxx> on 2019-01-20T00:00:00+00:00
[2019-01-27 12:43:56,581] {base_task_runner.py:118} INFO - Running: ['bash', '-c', 'airflow run xxx xxx 2019-01-20T00:00:00+00:00 --job_id 15 --raw -sd DAGS_FOLDER/xxx.py --cfg_path /tmp/tmpxx39wldz']
[2019-01-27 12:45:21,603] {models.py:1355} INFO - Dependencies not met for <TaskInstance: xxx.xxx 2019-01-20T00:00:00+00:00 [running]>, dependency 'Task Instance Not Already Running' FAILED: Task is already running, it started on 2019-01-27 12:43:56.565328+00:00.
[2019-01-27 12:45:21,639] {models.py:1355} INFO - Dependencies not met for <TaskInstance: xxx.xxx 2019-01-20T00:00:00+00:00 [running]>, dependency 'Task Instance State' FAILED: Task is in the 'running' state which is not a valid state for execution. The task must be cleared in order to be run.
[2019-01-27 12:45:21,641] {logging_mixin.py:95} INFO - [2019-01-27 12:45:21,641] {jobs.py:2614} INFO - Task is not able to be run

有什么具体的我应该做才能将 pod 的状态返回到 Airflow 吗？的KubernetesPodOperator定义如下：

[2019-01-27 12:43:56,580] {models.py:1595} INFO - Executing <Task(KubernetesPodOperator): xxx> on 2019-01-20T00:00:00+00:00
[2019-01-27 12:43:56,581] {base_task_runner.py:118} INFO - Running: ['bash', '-c', 'airflow run xxx xxx 2019-01-20T00:00:00+00:00 --job_id 15 --raw -sd DAGS_FOLDER/xxx.py --cfg_path /tmp/tmpxx39wldz']
[2019-01-27 12:45:21,603] {models.py:1355} INFO - Dependencies not met for <TaskInstance: xxx.xxx 2019-01-20T00:00:00+00:00 [running]>, dependency 'Task Instance Not Already Running' FAILED: Task is already running, it started on 2019-01-27 12:43:56.565328+00:00.
[2019-01-27 12:45:21,639] {models.py:1355} INFO - Dependencies not met for <TaskInstance: xxx.xxx 2019-01-20T00:00:00+00:00 [running]>, dependency 'Task Instance State' FAILED: Task is in the 'running' state which is not a valid state for execution. The task must be cleared in order to be run.
[2019-01-27 12:45:21,641] {logging_mixin.py:95} INFO - [2019-01-27 12:45:21,641] {jobs.py:2614} INFO - Task is not able to be run

编辑：看起来基本容器已经完成，但airflow-xcom-sidecar仍在运行。有什么具体的我应该做来阻止那个吗？

Answer 1

如果不查看您的设置，很难准确判断，但看起来 pod 已完成，它正在尝试将xcom推送到您的主气流，但无法连接。我会检查日志airflow-xcom-sidecar。就像是：

$ kubectl logs <airflow-job-pod> -c airflow-xcom-sidecar

您还可以尝试使用以下KubernetesOperator命令运行xcom_push=False：

do_something = KubernetesPodOperator(
    task_id='xxx',
    image='gcr.io/project/image',
    namespace='default',
    name='xxx',
    arguments=['dummy'],
    xcom_push=False,
    in_cluster=True,
    image_pull_policy='Always',
    trigger_rule='dummy',
    dag=dag,
)