Airflow Dag文件夹-如何忽略笔记本检查点

Question

气流太聪明了，试图在jupyter笔记本检查点文件夹“ dags / .ipynb_checkpoints /”中拾取损坏，这将引发错误。

有没有一种方法可以配置气流以忽略特定模式的文件夹？就像我会.gitignore吗？

谢谢

Answer 1

You can create .airflowignore in dags folder:

.ipynb_checkpoints

From the docs:

A .airflowignore file specifies the directories or files in DAG_FOLDER that Airflow should intentionally ignore. Each line in .airflowignore specifies a regular expression pattern, and directories or files whose names (not DAG id) match any of the patterns would be ignored (under the hood, re.findall() is used to match the pattern). Overall it works like a .gitignore file.

.airflowignore file should be put in your DAG_FOLDER. For example, you can prepare a .airflowignore file with contents

project_a
tenant_[\d]

Then files like project_a_dag_1.py, TESTING_project_a.py, tenant_1.py, project_a/dag_1.py, and tenant_1/dag_1.py in your DAG_FOLDER would be ignored (If a directory’s name matches any of the patterns, this directory and all its subfolders would not be scanned by Airflow at all. This improves efficiency of DAG finding).

The scope of a .airflowignore file is the directory it is in plus all its subfolders. You can also prepare .airflowignore file for a subfolder in DAG_FOLDER and it would only be applicable for that subfolder.

Answer 2

将一个文件放置.airflowignore在您希望 Airflow 忽略的目录中。