我在两个 AWS EC2 实例上运行的集群环境中运行 Airflow。一份给主人,一份给工人。不过,工作节点在运行“$airflowworker”时会定期抛出此错误:
[2018-08-09 16:15:43,553] {jobs.py:2574} WARNING - The recorded hostname ip-1.2.3.4 does not match this instance's hostname ip-1.2.3.4.eco.tanonprod.comanyname.io
Traceback (most recent call last):
File "/usr/bin/airflow", line 27, in <module>
args.func(args)
File "/usr/local/lib/python3.6/site-packages/airflow/bin/cli.py", line 387, in run
run_job.run()
File "/usr/local/lib/python3.6/site-packages/airflow/jobs.py", line 198, in run
self._execute()
File "/usr/local/lib/python3.6/site-packages/airflow/jobs.py", line 2527, in _execute
self.heartbeat()
File "/usr/local/lib/python3.6/site-packages/airflow/jobs.py", line 182, in heartbeat
self.heartbeat_callback(session=session)
File "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py", line 50, in wrapper
result = func(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/airflow/jobs.py", line 2575, in heartbeat_callback
raise AirflowException("Hostname of …Run Code Online (Sandbox Code Playgroud) 问题:我想apache-airflow在Github上使用最新版本的Apache-Airflow以及所有依赖项进行安装吗?
我该如何使用pip?
在生产环境中使用它也安全吗?
我有一个使用Airflow版本1.9的Airflow环境,该环境在Amazon EC2实例上运行.我需要升级到Airflow的最新版本1.10.我可以选择从1.9版升级或在新服务器上新安装1.10.气流版本1.10未在Pip上列出,所以我通过此命令从Git安装它,
pip-3.6 install git+git://github.com/apache/incubator-airflow.git@v1-10-stable
Run Code Online (Sandbox Code Playgroud)
此命令成功安装Airflow版本1.10.您可以通过运行命令airflow version并查看输出来查看
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
v1.10.0
Run Code Online (Sandbox Code Playgroud)
当我尝试启动Airflow调度程序时,airflow scheduler我得到以下异常,
ModuleNotFoundError: No module named 'MySQLdb'
[2018-08-14 14:03:16,195] {celery_executor.py:112} ERROR - Error syncing the celery executor, ignoring it:
[2018-08-14 14:03:16,195] {celery_executor.py:113} ERROR - …Run Code Online (Sandbox Code Playgroud)