损坏的 DAG：没有名为“airflow.contrib.gsc_to_gcs”的模块

Question

Airflow/Python 等非常新，但似乎无法弄清楚我需要做什么来解决这个问题..

Airflow 在 Puckel/Docker 上运行

完整的错误是：

Broken DAG : [/usr/local/airflow/dags/xxxx.py] No module named 'airflow.contrib.operators.gsc_to_gcs'

在 python 代码中，我写了：

from airflow.contrib.operators.gcs_to_gcs import GoogleCloudStorageToGoogleCloudStorageOperator

我猜我需要安装该gcs_to_gcs模块，但我不确定如何执行此操作。

任何具体说明将不胜感激:-)

Answer 1

该GoogleCloudStorageToGoogleCloudStorageOperator所以你必须从文件中复制不在v1.9.0可用在这里和相关的钩从这里在各自的位置，并将其粘贴在气流文件夹中的Python环境。请按照以下步骤操作：

运行以下代码以查找 Apache Airflow 在您的计算机上的存储位置：

pip show apache-airflow

这应该在您的终端上产生以下输出：

Name: apache-airflow
Version: 2.0.0.dev0+incubating
Summary: Programmatically author, schedule and monitor data pipelines
Home-page: http://airflow.incubator.apache.org/
Author: Apache Software Foundation
Author-email: dev@airflow.incubator.apache.org
License: Apache License 2.0
Location: /Users/kaxil/anaconda2/lib/python2.7/site-packages
Requires: iso8601, bleach, gunicorn, sqlalchemy-utc, markdown, flask-caching, alembic, croniter, flask-wtf, requests, tabulate, psutil, jinja2, gitpython, python-nvd3, sqlalchemy, dill, flask, pandas, pendulum, flask-login, funcsigs, flask-swagger, flask-admin, lxml, python-dateutil, pygments, werkzeug, tzlocal, python-daemon, setproctitle, zope.deprecation, flask-appbuilder, future, configparser, thrift
Required-by:

Location:后面的路径是你的Apache Airflow目录

现在克隆 git repo 以获取这两个文件：

# Clone the git repo to `airflow-temp` folder
git clone https://github.com/apache/incubator-airflow airflow-temp

# Copy the hook from the cloned repo to where Apache Airflow is located
# Replace LINK_TO_SITE_PACKAGES_DIR with the path you found above
cp airflow-temp/airflow/contrib/hooks/gcs_hook.py LINK_TO_SITE_PACKAGES_DIR/airflow/contrib/hooks/

# For example: for me, it would be 
cp airflow-temp/airflow/contrib/hooks/gcs_hook.py /Users/kaxil/anaconda2/lib/python2.7/site-packages/airflow/contrib/hooks/

# Do the same with operator file
cp airflow-temp/airflow/contrib/operators/gcs_to_gcs.py LINK_TO_SITE_PACKAGES_DIR/airflow/contrib/operators/

# For example: for me, it would be 
cp airflow-temp/airflow/contrib/operators/gcs_to_gcs.py /Users/kaxil/anaconda2/lib/python2.7/site-packages/airflow/contrib/operators/

重新运行 Airflow webserver，scheduler现在应该可以工作了。