运行`airflow scheduler`启动33个调度程序进程

Question

当LocalExecutor与MySQL后端一起使用时,airflow scheduler在我的Centos 6盒子上运行会创建33个调度程序进程,例如, deploy 55362 13.5 1.8 574224 73272 ? Sl 18:59 7:42 /usr/local/bin/python2.7 /usr/local/bin/airflow scheduler deploy 55372 0.0 1.5 567928 60552 ? Sl 18:59 0:00 /usr/local/bin/python2.7 /usr/local/bin/airflow scheduler deploy 55373 0.0 1.5 567928 60540 ? Sl 18:59 0:00 /usr/local/bin/python2.7 /usr/local/bin/airflow scheduler ... 这些进程与Executor进程和gunicorn主进程和工作进程不同.使用SequentialExecutor (sqlite后端)运行它只需启动一个调度程序进程.
气流仍然有效(DAG正在运行),但这些过程的绝对数量让我觉得有些不对劲.
当我select * from job where state = 'running';在数据库中运行时,只SchedulerJob返回5 行.这是正常的吗？

Answer 1

是的,这是正常的.这些是调度程序进程.您可以使用airflow.cfg中的以下参数来控制它

# The amount of parallelism as a setting to the executor. This defines
# the max number of task instances that should run simultaneously
# on this airflow installation
parallelism = 32

这些是从调度程序生成的,调度程序的pid可以在airflow-scheduler.pid文件中找到

所以你看到32 + 1 = 33个进程.

希望这能清除你的怀疑.

干杯!