相关疑难解决方法(0)

如何在Apache Airflow中同时使用Celery Executor和Kubernetes Executor?

我使用Celery Executor有多个dag,但是我想使用Kubernetes Executor运行一个特定的dag。我无法推断出实现此目标的好方法。

我已经airflow.cfg声明CeleryExecutor要在其中使用。而且我不想更改它,因为除了一只狗以外,其他所有狗狗都确实需要它。

# The executor class that airflow should use. Choices include
# SequentialExecutor, LocalExecutor, CeleryExecutor
executor = CeleryExecutor
Run Code Online (Sandbox Code Playgroud)

我的问题代码:

from datetime import datetime, timedelta

from airflow import DAG
from airflow.contrib.operators.kubernetes_pod_operator import \
    KubernetesPodOperator
from airflow.operators.dummy_operator import DummyOperator

default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'start_date': datetime.utcnow(),
    'email': ['airflow@example.com'],
    'email_on_failure': False,
    'email_on_retry': False,
    'retries': 1,
    'retry_delay': timedelta(minutes=5)
}

dag = DAG(
    'kubernetes_sample_1', default_args=default_args)


start = DummyOperator(task_id='run_this_first', dag=dag)

passing = KubernetesPodOperator(namespace='default',
                                image="Python:3.6",
                                cmds=["Python", "-c"],
                                arguments=["print('hello …
Run Code Online (Sandbox Code Playgroud)

python celery python-3.x kubernetes airflow

5
推荐指数
1
解决办法
175
查看次数

标签 统计

airflow ×1

celery ×1

kubernetes ×1

python ×1

python-3.x ×1