气流随机向任务发送sigterms

Question

我遇到了气流 1.10.1 的问题。SIGTERMdag中的一些任务来自 helpers.py，据我所知，这是为工作人员执行关闭并终止所有子进程，但我只在 10 个 dag taks 中的大约 2-3 个任务中看到了这一点，并且再次运行 dag 时，接收信号的任务会发生变化。是否有一定的标准来发送这些SIGTERM信号。收到 SIGTERM 的任务的日志：

[2019-12-10 11:13:44,530] {base_task_runner.py:101} INFO - Job 25404: Subtask BS_PMU2 [2019-12-10 11:13:44,520] {settings.py:174} INFO - setting.configure_orm(): Using pool settings. pool_size=5, pool_recycle=3600
[2019-12-10 11:13:45,489] {base_task_runner.py:101} INFO - Job 25404: Subtask BS_PMU2 [2019-12-10 11:13:45,488] {__init__.py:51} INFO - Using executor CeleryExecutor
[2019-12-10 11:13:45,934] {base_task_runner.py:101} INFO - Job 25404: Subtask BS_PMU2 [2019-12-10 11:13:45,933] {models.py:271} INFO - Filling up the DagBag from /home/centos/airflow/dags/61b6c300e82643b0f294df6f.py
[2019-12-10 11:13:46,580] {base_task_runner.py:101} INFO - Job 25404: Subtask BS_PMU2 Connected to MongoDB...
[2019-12-10 11:13:47,510] {bash_operator.py:74} INFO - Tmp dir root location:
/tmp
[2019-12-10 11:13:47,510] {bash_operator.py:87} INFO - Temporary script location: /tmp/airflowtmpal71kawr/BS_PMU2rjty_k9l
[2019-12-10 11:13:47,511] {bash_operator.py:97} INFO - Running command:
[2019-12-10 11:13:47,542] {bash_operator.py:106} INFO - Output:
[2019-12-10 11:13:47,542] {bash_operator.py:114} INFO - Command exited with return code 0
[2019-12-10 11:13:57,559] {base_task_runner.py:101} INFO - Job 25404: Subtask BS_PMU2 2019-12-10 11:13:57,556 - root - INFO - Putting xcom with return value:
[2019-12-10 11:13:57,631] {base_task_runner.py:101} INFO - Job 25404: Subtask BS_PMU2 2019-12-10 11:13:57,625 - root - INFO - WorkflowID: 61b6c300e82643b0f294df6f, RunID: 456c5bfb16556a3adc3b251a, TaskID: BS_PMU2
[2019-12-10 11:13:57,652] {base_task_runner.py:101} INFO - Job 25404: Subtask BS_PMU2 2019-12-10 11:13:57,643 - root - ERROR - Invalid key/value. Will skip setting xcom.
[2019-12-10 11:13:57,652] {base_task_runner.py:101} INFO - Job 25404: Subtask BS_PMU2 2019-12-10 11:13:57,644 - root - INFO - Done Execute
[2019-12-10 11:13:58,663] {helpers.py:240} INFO - Sending Signals.SIGTERM to GPID 9696
[2019-12-10 11:13:58,674] {helpers.py:230} INFO - Process psutil.Process(pid=9696 (terminated)) (9696) terminated with exit code 15```

Answer 1

AIRFLOW__CORE__KILLED_TASK_CLEANUP_TIME如果您想坚持使用相同版本的气流，可​​以尝试增加气流配置中的值。

升级您的气流版本 >= 2.X 也会有所帮助。

您可以查找文档以获取更多参考：https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#killed-task-cleanup-time