气流系统调度程序无法正常工作

Question

我让网络服务器正常工作,我的airflow-scheduler.service文件启动调度程序,它找到了我的dags等.但是,任务没有运行:

我看到有关/ bin/sh的错误消息

ERROR - failed to execute task Command 'exec bash -c run'

我有我的sysconfig文件:

#!/bin/bash
PATH=/opt/anaconda/anaconda3/envs/airflow_env/bin/airflow
AIRFLOW_CONFIG=/mnt/var/airflow/airflow.cfg
AIRFLOW_HOME=/mnt/var/airflow

还有我的airflow-scheduler.service文件:

#!/bin/bash
[Unit]
Description=Airflow scheduler daemon
After=network.target postgresql.service
Wants=postgresql.service

[Service]
EnvironmentFile=/etc/sysconfig/airflow
User=airflow
Group=airflow
Type=simple
ExecStart=/opt/anaconda/anaconda3/envs/airflow_env/bin/airflow scheduler
Restart=always
RestartSec=5s

[Install]
WantedBy=multi-user.target

这是记录我得到的bash错误的journalctl记录:

[2017-10-30 18:36:13,764] {base_executor.py:50} INFO - Adding to queue: airflow run user_presence_raw_etl transform_raw_user_presence 2017-10-30T14:00:00 --local -sd /mnt/var/airflow/dags/bin/user_p
Oct 30 18:36:13  airflow[4742]: [2017-10-30 18:36:13,765] {jobs.py:1443} INFO - Heartbeating the executor
Oct 30 18:36:13  airflow[4742]: [2017-10-30 18:36:13,783] {local_executor.py:45} INFO - LocalWorker running airflow run user_presence_raw_etl transform_raw_user_presence 2017-10-30T14:00:00 --local -sd /mnt/var/airflow/dags/bin/us
Oct 30 18:36:13  airflow[4742]: /bin/sh: 1: exec: bash: not found
Oct 30 18:36:13  airflow[4742]: [2017-10-30 18:36:13,865] {local_executor.py:52} **ERROR - failed to execute task Command 'exec bash -c 'airflow run** user_presence_raw_etl transform_raw_user_presence 2017-10-30T14:00:00 --local -sd /mnt/var/airf
Oct 30 18:36:14  airflow[4742]: [2017-10-30 18:36:14,786] {jobs.py:1407} INFO - Heartbeating the process manager
Oct 30 18:36:14  airflow[4742]: [2017-10-30 18:36:14,786] {dag_processing.py:559} INFO - Processor for /mnt/var/airflow/dags/bin/prod/hourly_agent_dag.py finished
Oct 30 18:36:14  airflow[4742]: [2017-10-30 18:36:14,789] {dag_processing.py:627} INFO - Started a process (PID: 5425) to generate tasks for /mnt/var/airflow/dags/bin/prod/daily_agent_email_dag.py - logging into /mnt/var/airflow/l
Oct 30 18:36:14  airflow[4742]: [2017-10-30 18:36:14,831] {jobs.py:1000} INFO - No tasks to send to the executor
Oct 30 18:36:14  airflow[4742]: [2017-10-30 18:36:14,832] {jobs.py:1443} INFO - Heartbeating the executor
Oct 30 18:36:14  airflow[4742]: [2017-10-30 18:36:14,833] {jobs.py:1195} INFO - Executor reports user_presence_raw_etl.transform_raw_user_presence execution_date=2017-10-30 14:00:00 as failed

Answer 1

由 LocalExecutor 生成的工作进程无法运行气流，因为它在PATH. 您只需将其配置为sysconfig. 在气流存储库airflow-scheduler.service中提供的示例中，他们希望用户可以执行气流。

我的建议是为airflow创建virtualenv，在那里安装airflow，然后为每个你想要运行与airflow相关的东西的shell激活它，例如工作人员、调度程序或网络服务器。