相关疑难解决方法(0)

我正在使用docker-compose来设置可扩展的气流群集.我的解决方案基于这个Dockerfile https://hub.docker.com/r/puckel/docker-airflow/

我的问题是将日志设置为从s3写入/读取.当一个dag完成后,我得到这样的错误

*** Log file isn't local.
*** Fetching here: http://ea43d4d49f35:8793/log/xxxxxxx/2017-06-26T11:00:00
*** Failed to fetch log file from worker.

*** Reading remote logs...
Could not read logs from s3://buckets/xxxxxxx/airflow/logs/xxxxxxx/2017-06-
26T11:00:00

我在这个airflow.cfg文件中设置了一个新的部分

[MyS3Conn]
aws_access_key_id = xxxxxxx
aws_secret_access_key = xxxxxxx
aws_default_region = xxxxxxx

然后在远程日志部分中指定s3路径 airflow.cfg

remote_base_log_folder = s3://buckets/xxxx/airflow/logs
remote_log_conn_id = MyS3Conn

我是否正确设置了这个并且有错误？这里有成功的秘诀吗？

- 更新

我尝试以URI和JSON格式导出,似乎都不起作用.然后我导出了aws_access_key_id和aws_secret_access_key,然后气流开始捡起它.现在我在工作日志中得到了他的错误

6/30/2017 6:05:59 PMINFO:root:Using connection to: s3
6/30/2017 6:06:00 PMERROR:root:Could not read logs from s3://buckets/xxxxxx/airflow/logs/xxxxx/2017-06-30T23:45:00
6/30/2017 6:06:00 PMERROR:root:Could not write logs to s3://buckets/xxxxxx/airflow/logs/xxxxx/2017-06-30T23:45:00
6/30/2017 6:06:00 …

我试图在具有以下配置/版本的三个节点的开发 Hadoop 集群上安装和配置 apache Airflow：

Operating System: Red Hat Enterprise Linux Server 7.7\npython 3.7.3\nanaconda 2\nspark 2.45\n\na)sudo yum install gcc gcc-c++ -y\nb)sudo yum install libffi-devel mariadb-devel cyrus-sasl-devel -y\nc)pip install \'apache-airflow[all]\'\nd)airflow initdb  -- airflow.cfgfile was created with SQLlite\n

然后我按照下面的命令集使用 mysql 配置它

a) rpm -Uvh\xc2\xa0https://repo.mysql.com/mysql80-community-release-el7-3.noarch.rpm \nb) sed -i \'s/enabled=1/enabled=0/\' /etc/yum.repos.d/mysql-community.repo \nc) yum --enablerepo=mysql80-community install mysql-community-server \nd) systemctl start mysqld.service\n

在 mysql 中完成以下操作

a) CREATE DATABASE airflow CHARACTER SET utf8 COLLATE utf8_unicode_ci; \nb) create user \'airflow\'@\'localhost\' identified by \'Airflow123\'; …