每隔X分钟运行一次Airflow DAG

Question

我正在使用该LocalScheduler选项在EC2实例上使用气流.我已经调用了airflow scheduler,airflow webserver而且一切似乎都运行正常.也就是说,在将cron字符串提供schedule_interval给"每10分钟'*/10 * * * *'执行一次"之后,默认情况下,作业每24小时继续执行一次.这是代码的标题:

from datetime import datetime
import os
import sys

from airflow.models import DAG
from airflow.operators.python_operator import PythonOperator

import ds_dependencies

SCRIPT_PATH = os.getenv('PREPROC_PATH')

if SCRIPT_PATH:
    sys.path.insert(0, SCRIPT_PATH)
    import workers
else:
    print('Define PREPROC_PATH value in environmental variables')
    sys.exit(1)

default_args = {
  'start_date': datetime(2017, 9, 9, 10, 0, 0, 0), #..EC2 time. Equal to 11pm hora México
  'max_active_runs': 1,
  'concurrency': 4,
  'schedule_interval': '*/10 * * * *' #..every 10 minutes
}

DAG = DAG(
  dag_id='dash_update',
  default_args=default_args
)

...

Answer 1

default_args仅用于填充传递给DAG中的运算符的参数.max_active_runs,concurrency和,schedule_interval都是初始化DAG的参数,而不是运算符.这就是你想要的:

DAG = DAG(
  dag_id='dash_update',
  start_date=datetime(2017, 9, 9, 10, 0, 0, 0), #..EC2 time. Equal to 11pm hora México
  max_active_runs=1,
  concurrency=4,
  schedule_interval='*/10 * * * *', #..every 10 minutes
  default_args=default_args,
)

之前我也将它们混合在一起,以供参考(注意有重叠):

DAG参数:https://airflow.incubator.apache.org/code.html?highlight=dag#airflow.models.DAG 操作参数:https://airflow.incubator.apache.org/code.html#baseoperator

Answer 2

对于气流版本>2.1，您可以使用datetime.timedelta()对象：

DAG = DAG(
  dag_id='dash_update',
  start_date=datetime(2017, 9, 9, 10, 0, 0, 0),
  max_active_runs=1,
  concurrency=4,
  schedule_interval=timedelta(minutes=10),
  default_args=default_args,
)

处理 start_date 的另一个很酷的功能是days_ago

from airflow.utils.dates import days_ago

DAG = DAG(
  dag_id='dash_update',
  start_date=days_ago(2, minute=15), # would start 2 days ago at 00:15
  max_active_runs=1,
  concurrency=4,
  schedule_interval=timedelta(minutes=10),
  default_args=default_args,
)